Google Scholar should probably come with a health warning for academics and anyone looking at academics’ work. It is undeniably useful, but some aspects of it can be very misleading. That it has these shortcomings is possibly a product of a quantificationist attitude of its designers (or perhaps a lack of care?). Here are some things I’ve noticed while using it (both in the sense of browsing it and tracking my own citations):
- What is classed as a citation varies wildly in terms of type. A fully peer-reviewed paper at a top conference or in a respected journal counts the same as someone’s Masters thesis or even an academic-looking PDF that has been uploaded somewhere (and cites your work). This is not to disparage the quality of said Masters thesis or unpublished paper, but merely to point out that they really are different things that require entirely different ways of counting.
- Self-citation appears identical to other citation types.
- Exclusion and inclusion is opaque. It is generally unclear by what policy something gets indexed and therefore is classed as a citation and what doesn’t make the cut (and why).
- Citation ‘scores’ are not necessarily comparable across disciplines or even within a discipline. The fact that Google Scholar autogenerates a kind of research 'leaderboard’ from its data falsely implies otherwise. Firstly, some disciplines cite more than others. Secondly, the form of publication varies hugely. Some academics publish frequently in conferences (e.g., HCI) with short lead times to publication. Compare this with journals that may take years (and therefore slower-emerging citation rates). Clearly the temporality of different disciplines is unrepresented by such attempts at 'gamifying’ research citation metrics.
- Citation scoring tends to favour recent publications.