Saturday, 28 May 2016

EgaFem Analysis: Evaluation Criteria

Author: Blaise Wilson

As part of the EgaFem Analysis series we want to help improve the quality of debates by understanding not only how to judge evidence, but also how to write academic style reports.

The aim of a report is to persuade the reader.

A report is made of two elements, the facts/evidence and the way the report is written. The report could be amazing but based on faulty facts, or the facts could be perfect but the report could be badly presented. And every variant in between.

One way to learn is to study other examples to learn what to and not to do. This will help you learn what needs improving in your own writing, or spot other good and bad examples. In the future we may use this evaluation criteria to mark other reports, and compare them. To do this we need a standard to mark against.

Our first example will be an example of what not to do. The Cyberviolance Report [6] was so bad it has been retracted by the Broadband Commission. However it is still available on the UN Women’s website.

It is worth reading this article to know what ‘good’ looks like, then read the Cyberviolence report with those in mind. If you score each element out of 10 it will be fun to see where we agree and where we differ – and please leave a comment on this website, or tweet us @EgaFem, or contact us on facebook [7] with your own score for comparison.

We’ll be analysing the reports though reasoned, unbiased clear and fair repeatable criteria, judgement, and evidence. This is called an Evaluation Essay [1]. This is basically how you are marked at university.

The criteria is based off the marking scheme of my own university uses, tailored for this application and mixed with a few other sources. We will use a marking scheme so that future reports can also be judged and compared.

We’ll break down how the report will be analysed, then crack on with its application to the UN report.

Concise, logical and clearly laid out structure


A good report should be easy to read, follow through logically without jumping about and have a clearly laid out structure. The standard for this structure in academia is: Abstract, Introduction, Main Body (separated into smaller parts if needed), Discussion, Conclusions, References and Appendix. Not all reports have to prescribe to this template, but if they deviate it still needs to flow and makes sense.

Being concise is using as few words as you can possibly get away with in order to get a point across without having superfluous wording making the sentence long, boring and hard to read.

Conciseness is being economical with words.

Clear aims and boundary


A report needs to have clearly stated goals and scope, ideally very early on in the report. This orientates the reader and helps them understand what will and won’t be covered. This will also help identify out of scope or irrelevant topics. This is key to drawing conclusions and discussing how well the report achieves its goals.

Clearly defined terms


Jargon and specialised terminology needs to be clearly defined. As should any term that is open to interpretation or not be clearly bounded. This ensures the reader understands how the report interprets the wording. Failure to do this could result in miss-understanding and assumptions being made by all, which may not agree. The reader does not need to agree with the definition, simply understand how the report is using certain words.

The definition needs to remain consistent throughout the report.

Wider topics and contextual understanding


Orientating the reader and making it clear how the report fits into the bigger picture is very important. This is highly linked to the aims, scope and boundary of the report. It should be clear how the report is dependent on outside influences, and how the wider context and linked topics could be impacted by the report.

Any recommendations and solutions should discuss the impact on the wider context using both positive and negative perspectives.

Critical analysis of data


Critical analysis of data is a key point in academia. Without this the report may be accused of being biased and having an agenda. This puts the report at risk of being labelled propaganda [3].

Evidence needs to be provided. Without a provided citation the statement is an assumption or assertion. Statements without evidence can be dismissed without evidence. In this kind of analysis the quality of the evidence is not in question, simply if evidence is or is not provided.

Another important part of academic writing is to provide a counter-argument to the premise and explore alternative views. If this is not done then the report could be biased.

This is particularly evident if the views put forward are not analysed in terms of positives and negatives. If only positives points or no analysis is done at all the report is being biased.

Using words like ‘clearly’ and ‘obviously’ are good indicators of poor analysis.

Other symptoms of bias are [4]:
  • Extreme language; statements have all or nothing connotations.
  • The argument appeals more to the emotions than to logic.
  • Things are worded with the intent to oversimplify or over generalize.
  • A limited view of the topic.
A presumption of a solutions, without seeking alternatives or even analysing the effectiveness of the solution is also biased. A solution should be found after identifying what it is trying to achieve and then analysis if this solutions will achieve those aims. Ideally discussing multiple solutions and picking the best from the analysis.

These solutions should also be analysed by the impact and implications on the wider context.

Logical fallacies can also mire a report. There are many of these fallacies, the most common ones are highlighted at: https://yourlogicalfallacyis.com [2]. A collection of logical fallacies linked to spotting propaganda can be found here: http://www.cuesta.edu/student/servs_classes/ssc/study_guides/critical_thinking/103_think_logic_errors.html [5]

Discussion


The report should analysis itself, discussing assumptions and judgements made during the report. This should include highlighting the fidelity of any conclusions and discussing problems with the methodology. It should also suggest further work that needs to be done to improve the results.


Draw Appropriate Conclusions


The report needs to draw appropriate conclusions, based on the arguments put forth within the report. It should not pull in additional supporting evidence, but be based on the information previously analysed. It should be support by the raw data and analysis.

A Good Report with Marking Scheme


In summary a good report should:
  • Be concise, logical and be clearly laid out
  • Have clear aims and boundary
  • Have clearly defined consistent terms
  • Link to wider topics and context
  • Provide evidence
  • Provide a counter-argument to premise and explore alternative views
  • Link arguments and solutions to wider context, including implications
  • Avoid logical fallacies
  • Have a discussion
  • Draw appropriate, supported conclusions
Each element will be marked out of 10, making a total of 100 marks.

Marks will be awarded from a bottom up approach. In other words start at zero and the report will have to earn the points.

Evaluation Essay Marking Scheme
Criteria
Mark
Max Mark
Have clear aims and boundary

10
Have clearly defined consistent terms, tone and use of language

10
Providing evidence

10
Avoiding logical fallacies

10
Link main premise to wider topics and context

10
Providing a counter-argument to premise and explore alternative views

10
Linking arguments and solutions to wider context, including implications

10
Have a discussion

10
Draw appropriate, supported conclusions

10
Be concise, logical and be clearly laid out

10


Total
100

No comments:

Post a Comment