Coalescence times for three genes provide sufficient information to distinguish population structure from population size changes

Simona Grusea (1, 2) , Willy Rodríguez (1, 2) , Didier Pinchon (1) , Lounès Chikhi (3) , Simon Boitard (4) , Olivier Mazet (1, 2)

1 IMT - Institut de Mathématiques de Toulouse UMR5219
2 INSA Toulouse - Institut National des Sciences Appliquées - Toulouse
3 EDB - Evolution et Diversité Biologique
4 GenPhySE - Génétique Physiologie et Systèmes d'Elevage

The increasing amount of genomic data currently available is expanding the horizons of population genetics inference. A wide range of methods have been published allowing to detect and date major changes in population size during the history of species. At the same time, there has been an increasing recognition that population structure can generate genetic data similar to those generated under models of population size change. Recently, Mazet introduced the idea that, for any model of population structure, it is always possible to find a panmictic model with a particular function of population size change having an identical distribution of T_2 (the time of the first coalescence for a sample of size two). This implies that there is an identifiability problem between a panmictic and a structured model when we base our analysis only on T_2. A natural question that deserves to be explored is whether and when this identifiability problem disappears for larger sample sizes. In this paper, based on an analytical study of the rate matrix (or Q-matrix) of the ancestral lineage process, we obtain new theoretical results about the joint distribution of the coalescence times (T_3,T_2) for a sample of three haploid genes in a $n$-island model with constant size. In particular, we show that this distribution is always different from the analogous one obtained in a panmictic population, for any scenario of population size-change. Even if, for any k >= 2, it is always possible to find a size-change scenario for a panmictic population such that the marginal distribution of T_2 is exactly the same as in a $n$-island model with constant population size, we show that the joint distribution of the coalescence times (T_3,T_2) for a sample of three genes contains enough information to distinguish between a panmictic population and a n-island model of constant size.

Mots clés: population size change; demographic history; population structure; IICR (inverse instantaneous coalescence rate); rate matrix; structured coalescent.

Domaines: Mathématiques générales [math.GM]; Génétique des populations [q-bio.PE]

Simona Grusea, Willy Rodríguez, Didier Pinchon, Lounès Chikhi, Simon Boitard, et al.. Coalescence times for three genes provide sufficient information to distinguish population structure from population size changes. Journal of Mathematical Biology, 2018, ⟨10.1007/s00285-018-1272-4⟩. ⟨hal-01631938⟩