The Riddle of Literary Quality

In search of conventions of literariness:
The Riddle of Literary Quality

〈1〉 When the research project ‘The Riddle of Literary Quality’ was announced on the Huygens ING website, there was quite a stir in Dutch literary studies. I hope to show that this stir may be seen as a sign that this DH project touches upon issues that are fundamental in the way Humanities projects usually are conducted. I’ll start with a translation of the complete press release that was sent out on 7 July 2011, then summarize the first reactions, and then will continue with a more detailed description of the way the project and how we talk about it has developed over the last five years, to end with some remarks about the questions our meeting is dealing with.

〈2〉 “Literary scholars are in agreement that the quality of literary texts is established in the communication between readers and critics. Hardly any research has been done into which criteria could play a role in this. Which characteristics should a novel posess to be called highly literary or not literary at all? And what makes a novel, of whatever level of literariness, good or bad? Together with the Fryske Akademy and the Universiteit van Amsterdam, Huygens ING will approach these questions in a totally new way: by means of computational analysis of formal patterns in novels. This research is funded from the Computational Humanities programme of the KNAW (the Royal Netherlands Academy of Arts and Sciences).

〈3〉 Based on a large survey of readers, the project will search for those formal features that are shared by novels that the majority of the respondents indicate to find (highly) literary or not so very literary. Novels that are indicated as being found very good or bad will also be analysed. The underlying assumption for this is that ‘literary’ novels can also be bad, and that novels not described as ‘literary’ can indeed be good. Then, advanced software will be applied to establish which formal characteristics these four types of novels have: how varied is their vocabulary, what is the percentage of different parts of speech, for instance adjectives or adverbs, how long are the sentences and how complex is the story line?

〈4〉 The result of this analysis is an overview of the combination of characteristics in each of the four types: literary/good, literary/bad, not literary/good, not literary/bad. Next, features of the novels that according to the respondents of the survey fall in between these four groups will be analyzed. For each of these novels it will become clear to what amount they share features with each of the four main categories. All this could results in the identification of possible formal criteria that may have influenced readers in their evaluation of novels. Undoubtedly, new research questions will also arise.

〈5〉 For this research project new tools will be developed that after the end of the project will be made freely available for other researchers, making it possible to verify and repeat the research.

〈5〉 The project “The riddle of literary quality” will start in the Fall of 2011. Contactperson is Karina van Dalen-Oskam.”

〈6〉 The small computational humanities programme of the KNAW in which the project was funded was actively asking for experimental, daring, and innovative Digital Humanities projects that would focus on higher-level pattern recognition. The Riddle proposal was one of the four selected for funding because of the innovative combination of methods from different research traditions from both the Humanities and the Social Sciences and the intriguing research question.

〈7〉 The first reactions to our press release showed that literary scholars felt that the proposed research was impossible. What makes a novel literary, people stated, is different for each individual reader. Also, a reader’s reaction depends on how a publisher puts a novel in the market and how critics react to it – a purely sociological process. Another reflex was to state that not only readers, but also literary works are unique, so looking for shared patterns is destined to fail. Another underlying fear to the outraged reactions seemed to be that this project showed there actually were scholars who thought that the computer could take over the hermeneutic task of the literary scholar and that the core qualities of humanities research were not being taken seriously.

〈8〉 These were the reactions out in the open. We also received private communications, from scholars wishing us luck and complimenting us for our brave new approach, sending us their PhD-theses on related topics, and telling us they never understood why mainstream literary studies in the Netherlands has never been open to this kind of empirical approach. And many people from other academic disciplines and from outside of academia immediately showed their enthusiasm for the project and even wondered why we presented it as new: hadn’t this been done a long time ago, then?

〈9〉 All these extremely diverse reactions were very useful for us. The first big milestone in the project was the survey of readers’ opinions. Because the reactions to our project showed that the sociological perspective on literary quality was even more prominent and mainstream in Dutch literary studies than we has estimated beforehand, we wanted to make sure that the incoming data could also be used to study these aspects of literary evaluation. This meant the survey had to become much more elaborate than I had thought when writing the proposal. The initial idea was that to avoid deciding ourselves which novels were literary and which not and then analyze those, we would in fact delegate this decision to the respondents to our survey. But to make sure we would not make beginners’ mistakes in setting up the survey and thus reduce the usefulness of the survey’s outcomes, we decided to hire an experienced market researcher, the economic psychologist Erica Nagelhout. She helped us to set up the survey and we learned a lot from her. We realized that to make sure the survey results would also convince our critical audience, we would need as many respondents as possible. We calculated we needed about 18.000 people to start – taking into account a certain percentage would never finish the whole survey. This lead to another new element in our project: we hired a public relations specialist, who wrote and implemented a communications plan for us and who managed to arrange several interviews in newspapers and on radio shows. The Huygens communications department had already designed the project’s logo’s, and now sugested an intriguing name for the survey: The National Reader’s Survey. They designed a bookmark with the URL of the survey, and a set of 150 bookmarks was sent to all public libraries in the Netherlands. All these steps helped us to indeed getting enough respondents who completed the survey. We now have the results of a total of 13.782 respondents in our database.

〈10〉 From the start of the project, we also made use of a project blog with occasional updates about the project and the complete proposal ( We did the same for the survey ( After the close of the survey, we used the start page for occasional blogs with some first results. One of our advisers suggested that when speaking to colleagues from literary studies we would evade the use of the word ‘literary’ and the word ‘quality’. What we in fact were dealing with was the analysis of contemporary conventions of literariness. We took that advice to heart as well. From that moment on, we gave more and more talks about the project, and integrated it in our teaching.

〈11〉 In my yearly introductory course to computational literary studies at the University of Amsterdam I explain that one of the reasons for the existence of the project is the fact that more and more literary texts become available in digital form, and that we can do more with these than just searching and finding. But when we want to make active use of more data, we also need to enlarge our set of methods. We are almost forced to add some kind of quantitative approach. And these in their turn lead to the need for another way of evaluating results, and the chance to make better testable comparisons between texts and groups of texts. A new way of confirming or falsifying hypotheses comes into existence. It makes use of ways of reasoning from other disciplines. Using a (relatively small) corpus of 400 novels (as The Riddle now does) is completely different from analysing three novels in depth and come up with a conclusion that is extremely difficult to verify by others. These types of conclusions are in fact the hypotheses of the kind of projects computational literary studies may deal with. To give the students a feel of what it is like to look at more than three novels at once, I give them a set of exercises with AntConc, an easy-to-use concordance programme that also has a lot of other interesting functions, such as keyword analysis and visualizations of occurrences throughout a text in bar plots. And I surprise them with Zipf’s law, showing how frequent ‘unimportant’ words are and how many ‘important’ words, dealing with the actual ‘content’ of a novel do not occur as often as they thought. And only after several of these eye-opening exercises they get the feeling that there really may be something to this kind of Digital Humanities research after all and that they even may be able to explain that to their peers who were skeptical and did not subscribe to this course. To help them in this, I wrote a short article for a literary journal run by students describing the prehistory of the Riddle project. An English translation is published on the project blg (

〈12〉 To sum up: knowing that the project The Riddle of Literary Quality would be interesting for a wide audience, we decided to do a lot of PR at the beginning of the project. This lead to a huge amount of reactions, that helped us to see where our special attention was needed to make our research results convincing for our own peers and society in general. It led to changes in our approach of several aspects of the project. We found out which steps in our reasoning needed to be made more explicit, and could actually test in our talks and teaching what kind of explanation in what level of detail worked and what did not. The topic of literary quality or better the conventions of literariness proved to be so interesting for students that they were willing to make an extra effort in trying to understand the new approaches in our project.

〈13〉 The Riddle of Literary Quality is now in its final phase. Three PhD-theses are currently being written, and a book for a general audience is planned, both in Dutch and in English. Of course the riddle of literary quality is not completely solved, but we definitely know a lot more, and have plenty of results for interesting publications and follow-up projects.

Diesen Blogbeitrag zitieren
Karina van Dalen-Oskam (2016, 5. Juli). The Riddle of Literary Quality. Digitalität. Abgerufen am 19. Mai 2024, von

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.

Suche in OpenEdition Search

Sie werden weitergeleitet zur OpenEdition Search