VoxCheck

Organization: VoxUkraine
Applicant: Olena Shkarpova
Assessor: Oleg Khomenok
Edits made by the organization after this assessment

IFCN Staff wrote:

Remarks and explanations to the External Assessment Sheet

VoxCheck did not publish articles over the period between 01/02 - 19/02 due to illness of its leader and project manager Olena Shkarpova, as well as the other VoxCheck employees. It was a harsh flu that did not allow us to work on a full potential. However, we did make daily monitorings for the Rating of Liars, released videos and fulfilled all our obligations within the projects that VoxCheck makes together with other media. For example, during this period we made a radio program on Hromadske Radio (see here, 14/02) and released fact-checks for the TV-show “Hra sliv” (see programs on 01/02, 07/02 and 15/02). During the next week after the mentioned period we published three articles (please, check here: one, two, three).

The Annual Report with the financial statement of VoxUkraine for the year 2016 can be found here. The report for the year 2017 should be prepared within the next few months.

The fact-checking methodology, correction policy and claim submission can be found here. We are ready to publish it on our web-site as soon as the IFCN and external assessor will approve it.

Regarding the Liars rating, in our opinion the methodology is accurate from the scientific and statistical point of view. Although the number of the statements for some politicians is relatively smaller than for others, in this case it is important that we take not some sample, but the "general population" of all statements of the specific politicians. So we believe that if we take all public fact-checkable statements, we do not have biases, which could occur if we only rated sample of the statements. The collection method for statements is consistent across time and across politicians, which also eliminates risks for other biases. So the number of statement, although varies a lot from politician to politician does not lead to biases in the evaluation. Of course some politicians may speak less overall, but we do not think it should impact their "lie index".

There can be a risk that some statements can be missed due to the fact that they are collected manually by people - even though we monitor all media appearances of the politicians. In any case, even if we randomly do not take some statements (which is doubtful, but we admit this can be the case), the randomness across population, applied to all politicians, ensures objectivity of the rating. This methodology was developed and approved by the Editorial Board of VoxUkraine as statistically meaningful and sensible.

Methodology

https://voxukraine.org/uk/voxcheck-f-a-q/

Comments from the assessor

Oleg Khomenok: VoxCheck did great job updating the site in accordance to the comments in assessment sheet. I checked the methodology description on the Google Drive and to my mind it is well written and ready for publication on the VoxCheck web-site.

Now with all updates VoxCheck is fully compliant and can join IFCN 


Conclusion and recommendations
on 15-Aug-2018 (3 months ago)

Oleg Khomenok wrote:

I think Voxcheck is compliant with most principles but should make some updates on their web-site to be accepted.

There are several issues that should be addressed by Voxcheck to be compliant and accepted. They are listed in the comments below each section of evaluation together with recommendations. In short there are:

In Criterion 1 there is a need to provide the explanation of the absense of the stories on the site during Jan 30-Feb 20, 2018.

In Criterion 4a there is a need to publish publish financial statements for 2016, 2017 and translate them into to Ukrainian and Russian

In Criterion 5a there is a need to publish the fact-checking methodology on the web-site.

In Criterion 5b there is a need to create direct link that will enable reader in 2 clicks to submit a statement or link to be fact-checked with explanation of what is fact-checkable and what is not.

In Criterion 6 there is a need to develop and update correction policy in the rules®s manual as well publish it on the web-site.

The Lie index https://voxukraine.org/longreads/lie-theory/index-en.html https://voxukraine.org/longreads/lie-theory/index-ua.html in general raises a question of comparative index methodology since comparing 56 statements of Poroshenko with 32 of Vilkul for example and creating a rating chart in percentage looks irrelevant. Some of the politicians were rated on 10 statements, at the same time President of Ukraine was rated on 56 statements.

The Lie index is published under Voxcheck rubric of the site but to my mind should be separate product not directly associated with the Voxcheck since it has a lot of distortions in the comparison based on unequal initial numbers of statements. Notably that this story has switched off commenting option, I believe it would get a lot of criticizing comments.

For further updates of the site here is a good example of methodology description and correction policy placed on the page https://www.factcheck.org/our-process/

Finally I would recommend acceptance of organisation after updates on the web-site and clear explanation of the pause in publishing in February of 2018 will be provided and separating Lie index from the Voxcheck page.

on 15-Aug-2018 (3 months ago)

Oleg Khomenok recommended Accept with edits


Section 1: Organization

Criterion 1a
Proof of registration
Evidence required: Please provide evidence that the signatory is a legally-registered organization set up exclusively for the purpose of fact-checking or the distinct fact-checking project of a recognized media house or research institution.

VoxUkraine
15-Aug-2018 (3 months ago) Updated: 3 months ago
Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

VoxUkraine is registered as non-profit non-government organisation under Reg.No at 40116636 the Unified Register of Legal Entities of Ukraine. The Voxcheck founders check with the OSINT tools showed no direct connections to any political party or politician via companies or NGOs. It has distinct fact-checking section on the web-site of the organisation.



done_all 1a marked as Fully compliant by Oleg Khomenok.

Criterion 1b
Archive
Evidence required: Insert a link to the archive of fact checks published in the previous three months. If you do not collect all fact checks in one place, please explain how the fact-checking is conducted by your organization.

VoxUkraine
15-Aug-2018 (3 months ago) Updated: 3 months ago
Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

Voxcheck regularly provided analysis and published reports in Ukrainian until the end of January 2018 but there is no reports for February 2018 at all with no explanation of the gap to the readers.


done 1b marked as Partially compliant by Oleg Khomenok.

Section 2: Nonpartisanship and Fairness

Criterion 2a
Body of work sample
Evidence required: Please share links to ten fact checks that better represent the scope and consistency of your fact-checking. Provide a short explanation of how your organization strives to maintain coherent standards across fact checks.

VoxUkraine
15-Aug-2018 (3 months ago) Updated: 3 months ago

1. We check all major politicians on a regular basis and publish checks on them. When we make weekly digests, we monitor all the speeches given by the politicians from the list (who are from all political parties represented in parliament) and take only those quotes that contain facts and figures.

2. All fact-checks are double-checked by our volunteers and VoxCheck team and go through the peer-review procedure, which means that at least 2 editors from the VoxUkraine Editorial Board read our fact-checks and give comments (also noted in the Policies and Procedures Manual). None article can be published until it is signed off by two editors.

3. Since the launch we've made more than 1300 fact-checks of Ukrainian politicians and some of the fact-checks were incoherent due to appearance of new data after that. Thus, in December 2017 we started making an audit of all fact-checks. The inconsistent fact-checks had been put under review and now they undergo a peer-review process.

Here are fact-checks of the President, the PM, the members of parliament and other politicians - both from the ruling parties and opposition:

1. A fact-check of the popular political talk-show and its guests - famous Ukrainian politicians from the major parties represented in the parliament. https://voxukraine.org/uk/svoboda-slova-vid-politikiv/

2. Analysis of the President of Ukraine fact-checked speeches over the period of 2 years: https://voxukraine.org/uk/pro-shho-breshe-petro-poroshenko-analiz-jogo-promov-za-dva-roki/

3. Analysis of the leader of the Opposition in Ukraine fact-checked speeches over the period of 2 years: https://voxukraine.org/uk/yuliya-timoshenko-antologiya-manipulyatsij/

4 Weekly digests of 17 top-politicians (here's an example: https://voxukraine.org/uk/pro-shho-lyublyat-rozpovidati-vibortsyam-poroshenko-ta-timoshenko-21-dajdzhest-voxcheck-impact-17/). For these digests, we took all the quotes that contained facts from all media sources - YouTube, Facebook, online-media, political parties web-sites etc. This work was made together with a team of VoxCheck volunteers (16 students from different universities in Ukraine)

5. Based on these digests we made a rating of liars: https://voxukraine.org/longreads/lie-theory/index-en.html

6. Top-5 fact-checked political myths of 2017 which were told by politicians to the people: https://voxukraine.org/uk/mifi-2017-pro-shho-najbilshe-brehali-politiki-protyagom-roku/

7. The fact-check of the conflict between the famous doctor and the Minister of Healthcare (both sides checked equally): https://voxukraine.org/uk/todurov-vs-ministerstvo-ua/

These articles include much more than 10 fact-checks. One can find all fact-checks here: https://voxukraine.org/uk/voxcheck/

Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

In general Voxcheck looks pretty good and provides extended and deep analysis of the politicians statements tracking representatives of various political forces and public events where some statements had been made.


done_all 2a marked as Fully compliant by Oleg Khomenok.

Criterion 2b
Nonpartisanship policy
Evidence required: Please share evidence of your policy preventing staff from direct involvement in political parties and advocacy organizations. Please also indicate the policy your organization has as a whole regarding advocacy and supporting political candidates.

VoxUkraine
15-Aug-2018 (3 months ago) Updated: 3 months ago

VoxUkraine (which is a organization for the VoxCheck project) is a non-partisan NGO. This principle is noted in the Policies and Procedures Manual of the organization. As a fact-checking project VoxCheck checks both sides of the political debates, which means that we've equally fact-checked the President Petro Poroshenko, the Prime-Minister Volodymyr Groysman, the opposition leaders Yulia Tymoshenko, Yury Boiko, Oleg Liashko etc. This can be seen on our web-site and from our data-base (access can be provided on demand).

VoxUkraine also demands from all editors, board members and employees to disclose any potential conflict of interest (Policies and Procedures Manual). Thus each article, including fact-checks, published on VoxUkraine web-site contains a disclaimer on potential conflict of interest or its absence.

Also, none of the members of the VoxUkraine governing bodies (General Meeting and Supervisory Board) are members of any political party.

Regarding hiring the people, who are members of political parties, we never had such a case since all our employees are not engaged into politics. However, we consider it will be very important to reflect this policy in Policies and Procedures Manual and not hire those candidates that are engaged into political activities. 

Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

Voxcheck defines statements on True, False, Manipulative and Exaggeration/Underestimation. True means either correct facts or data put in the relevant context. False means wrong or incomplete facts or data. Manipulative means correct facts used in irrelevant context that create distorted understanding of situation. Exaggeration/Underestimation means using correct facts and data but the influence on the understanding of situation is either exaggerated or underestimated.

The number of negative definitions is higher than positive (3 against 1).  


done_all 2b marked as Fully compliant by Oleg Khomenok.

Section 3: Transparency of Sources

Criterion 3a
Sources Policy
Please share a brief and public explanation (500 words max) of how sources are provided in enough detail that readers could replicate the fact check. If you have a public policy on how you find and use sources for your fact-checking, it should be shared here.

VoxUkraine
15-Aug-2018 (3 months ago) Updated: 3 months ago

VoxCheck uses only public official primary sources that can be shown on the web directly or, if not available, publishes the scans of the primary sources (info received from the authorities). In this way each fact-check can be double-checked by the readers by clicking on the links of the sources.

We refuse making fact-check and if we cannot find a public source as a proof. All the links are embeded in the fact-checks which can be seen on our web-site (for example here - a fact-check of the Prime Minister press conference: https://voxukraine.org/uk/za-rezultatami-roku-faktchek-pidsumkovoyi-press-konferentsiyi-volodimira-grojsmana/)

Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

Every single fact-check report has links to relevant sources so anybody can track and look for the extra data and understand which source was used to verify the statement.


done_all 3a marked as Fully compliant by Oleg Khomenok.

Section 4: Transparency of Funding & Organization

Criterion 4a
Funding Sources
Evidence required: Please link to the section where you publicly list your sources of funding (including, if they exist, any rules around which types of funding you do or don't accept), or a statement on ownership if you are the branch of an established media organization or research institution.

VoxUkraine
15-Aug-2018 (3 months ago) Updated: 3 months ago

VoxUkraine is a non-profit organization that aims at diversifying its fundraising sources. 1) According to its Policies and Procedures Manual, Vox Ukraine Funding from a single entity or the size of a single grant should not be greater than 33% of organization’s annual budget. 2) Funding from a single private individual should not be greater than 10% of organization’s annual budget. VoxUkraine uses the following fundraising sources: Donor grants. VoxUkraine raises project-aimed and institutional funds from donor organizations if funding conditions and potential projects help to implement VoxUkraine’s mission, which is to raise the level of economic discussion in Ukraine. The majority of financial committee members must approve the grant application submission VoxUkraine rejects or returns funds to donors (including those raised from both entities and private individuals) if such decision is made by the majority of financial committee members. If donor (private individual or entity), who financially supported VoxUkraine did not state the preference for anonymity, his name (or name of the entity) would be stated in the VoxUkraine’s annual financial report. If donor stated the preference for anonymity or did not want to reveal the amount of money donated, VoxUkraine would not disclose such information. Donors and editorial policy No donor shall intervene in the VoxUkraine editorial policy or influence VoxUkraine content in any way. If donor tries to influence VoxUkraine content or its other activities, members of Editorial Board must be notified. If the majority of editorial board members think that a donor has an undue influence on the editorial process, any member of editorial board or management of VoxUkraine, they shall consult Ethical Committee, who would make a decision on this matter.

Files Attached
description 4a - AR 2016-2018_VO... (293 KB)
Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

The latest financial report on the site is for 2015, and only in English that is reducing the audience only by English speakers.

Recommendation: publish financial statements for 2016, 2017 on the https://voxukraine.org/uk/about-us/; translate to Ukrainian and Russian https://voxukraine.org/uk/finansical-reports/


done 4a marked as Partially compliant by Oleg Khomenok.

Criterion 4b
Staff
Evidence required: Please link to the section detailing all authors and key actors behind your fact-checking project with their biographies. You can also list the name and bios of the members of the editorial board, pool of experts, advisory board, etc. if your organization has those.

VoxUkraine
15-Aug-2018 (3 months ago) Updated: 3 months ago

VoxUkraine employees (including VoxCheck) https://voxukraine.org/en/about-us-eng/#office VoxUkraine Editorial Board (also engaged in VoxCheck activities) https://voxukraine.org/en/about-us-eng/#board

Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

There are several ways readers can get in touch with the author or newsroom: comments under the story, links and buttons to the social networks’ accounts, “contact us” page. 


done_all 4b marked as Fully compliant by Oleg Khomenok.

Criterion 4c
Contact
Evidence required: Please link to the section where readers can get in touch with the organization.

VoxUkraine
15-Aug-2018 (3 months ago) Updated: 3 months ago
Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

The evidence submitted is fully compliant. 


done_all 4c marked as Fully compliant by Oleg Khomenok.

Section 5: Transparency of Methodology

Criterion 5a
Detailed Methodology
Evidence required: Please link to a section or article detailing the steps you follow for your fact-checking work.

VoxUkraine
15-Aug-2018 (3 months ago) Updated: 3 months ago

 Our methodology is written in the Manual for volunteers (Russian language): https://docs.google.com/document/d/1ypn_6jITNBabR962m8ouGl8A3hl3_x7uVZqQE7klHu0/edit Also, in the internal document Calendar, Rules and Definitions: https://docs.google.com/spreadsheets/d/1RAdiWMw_dIMLmdHTigQbHDx5v4RYz5vmPtbbiHhTSQQ/edit?usp=sharing. We also described the methodology in our Rating: https://voxukraine.org/longreads/lie-theory/index-en.html

Files Attached
picture_as_pdf 2b - POLICY AND PROC... (239 KB)
Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

the methodology is not published on the Voxcheck website as a step-by-step instruction on separate page with direct link. Brief explanation was once done for the Lie-index, but with no details. Recommendation: publish the methodology on the web-site.


done 5a marked as Partially compliant by Oleg Khomenok.

Criterion 5b
Claim submissions
Evidence required: Please link to the page or process through which readers can submit claims to fact-check. If you do not allow this, please briefly explain why.

VoxUkraine
15-Aug-2018 (3 months ago) Updated: 3 months ago

The claims are submitted via our Facebook page, personal messages or comments: https://www.facebook.com/VoxCheck/

Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

Despite the applicant states that claim can be sent “via our Facebook page, personal messages or comments”, it is unclear how readers can do it and there is no page indicating this.

Recommendation: create a button or item in the site index that will enable reader in 2 clicks to submit a statement or link to be fact-checked with explanation of what is fact-checkable and what is not. 


done 5b marked as Partially compliant by Oleg Khomenok.

Section 6: Open & Honest Corrections Policy

Criterion 6a
Corrections policy
Evidence required: Please link to the page with your policy to address corrections. If it is not public, please share your organization's handbook.

VoxUkraine
15-Aug-2018 (3 months ago) Updated: 3 months ago

This is not written in the Policies and Procedures Manual yet, but VoxCheck has repeatedly made corrections in the articles and admitted this before the public (see Criterion 6b). We will include this article into the Policy handbook soon. Also, independent editors reviewed some our debatable fact-checks and wrote their independent opinion (example: https://voxukraine.org/uk/perevirka-ua/)

Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

there is no step-by-step explanation of correction policy on the Voxcheck site as well as in the rules®s.

Recommendation: develop and update this policy in the rules®s manual as well publish it on the web-site. 


done 6a marked as Partially compliant by Oleg Khomenok.

Criterion 6b
Examples of corrections
Evidence required: Please provide two examples of a correction made, or correction requests handled, in the past year.

Oleg Khomenok Assessor
15-Aug-2018 (3 months ago) Updated: 3 months ago

The evidence submitted is sufficient. 


done_all 6b marked as Fully compliant by Oleg Khomenok.