© Wuhan University 2026

0 Introduction

Let's start with the following predator-prey model

$\{\begin{array}{l}{u}_t-\mathrm{\Delta }u=u(a(\mathrm{1}-ru)-\frac{n_{\mathrm{1}}}{u+n_{\mathrm{2}}})-p_{\mathrm{1}}uv,(x,t)\in \mathrm{\Omega }\times (\mathrm{0},\mathrm{\infty }),\\ v_t-\mathrm{\Delta }v=cv(\mathrm{1}-\frac{v}{u+k_{\mathrm{1}}}),\begin{array}{cc}& \end{array}(x,t)\in \mathrm{\Omega }\times (\mathrm{0},\mathrm{\infty }),\\ \frac{\partial u}{\partial n}=\frac{\partial v}{\partial n}=\mathrm{0},x\in \partial \mathrm{\Omega },t\in (\mathrm{0},\mathrm{\infty }),\\ u(x,\mathrm{0})=u_{\mathrm{0}}(x)\ge \mathrm{0},\cancel{\equiv }\mathrm{0},v(x,\mathrm{0})=v_{\mathrm{0}}(x)\ge \mathrm{0},\cancel{\equiv }\mathrm{0},\begin{array}{c}\end{array}x\in \overline{\mathrm{\Omega }},\end{array}$(1)

where $\mathrm{\Omega }\subset {\mathbb{R}}^n (n\ge \mathrm{1})$ is a bounded domain with smooth boundary $\partial \mathrm{\Omega }$. In Ref. [1], the authors studied the predator-prey model (1) and gave the conditions for the steady-state bifurcation and Hopf bifurcation from the unique positive constant solution. The $n_{\mathrm{1}}/(u+n_{\mathrm{2}})$ used here named additive Allee effect, where $n_{\mathrm{1}}$ and $n_{\mathrm{2}}$ are constants of Allee effect which describing the strength of Allee effect, refers to that the low density populations may have difficulties in finding mates, promoting reproduction, predation, environmental regulation and inbreeding, which may lead to population extinction. Therefore, it is significant to study the mechanism of Allee effect to avoid population extinction in low density population. Some classic studies on the impact of Allee effect on predation models can be found in Refs. [2-5]. The background and parameter meanings of this model can be found in Ref. [1] and we omit here.

It is well known that the predator-prey functional response is an important factor in predator-prey models, which profoundly affects the dynamics of predator-prey models^[6-7]. The functional response $p_{\mathrm{1}}u$ in (1) is linear function and is unbounded. This deficiency reminds us that we can use the following Ivlev function, which can be written as $\varphi (u)=I_{\mathrm{m}\mathrm{a}\mathrm{x}}(\mathrm{1}-{\mathrm{e}}^{-\gamma u}),$ where $u$ is the population density of prey, $\gamma$ and $I_{\mathrm{m}\mathrm{a}\mathrm{x}}$ are positive constants which represent the predation rate of the predator and the maximum capture rate, respectively. It is clear that the $\varphi (u)$ is bounded and satisfies ${\varphi }^{\text{'}}(u)=I_{\mathrm{m}\mathrm{a}\mathrm{x}}\gamma {\mathrm{e}}^{-\gamma u}>\mathrm{0}, {\varphi }^{″}(u)=-I_{\mathrm{m}\mathrm{a}\mathrm{x}}{\gamma }^{\mathrm{2}}{\mathrm{e}}^{-\gamma u}<\mathrm{0}, \underset{\gamma \to \mathrm{\infty }}{\mathrm{l}\mathrm{i}\mathrm{m}} \varphi (u)=I_{\mathrm{m}\mathrm{a}\mathrm{x}}.$

A recent experiment shows that the predation rate of Tortanus dextrilobatus expressed by $\gamma$ can simulate the model that Tortanus dextrilobatus prey on zooplankton in the San Francisco Estuary^[8]. Many studies, both modeling analysis and ecological experiments, such as Refs. [8-11], show that the predation rate $\gamma$ strongly affects the coexistence of predator and prey. However, there has been no research on using the Ivlev functional response to simulate predation of zooplankton in (1). For the sake of simplicity, we will take $I_{\mathrm{m}\mathrm{a}\mathrm{x}}=\mathrm{1}$ in our work. By introducing the following non-dimensional variables

$\overline{u}=aru, ar{n}_{\mathrm{1}}=m, ar{n}_{\mathrm{2}}=b, \frac{p_{\mathrm{1}}}{ar}=p, car=d, k_{\mathrm{1}}ar=k$

in (1) and dropping the superscripts of $u$ for simplicity, the predator-prey model (1) is given by the following equations

$\{\begin{array}{l}{u}_t-\mathrm{\Delta }u=u(a-u)-\frac{mu}{u+b}-(\mathrm{1}-{\mathrm{e}}^{-\gamma u})v,(x,t)\in \mathrm{\Omega }\times (\mathrm{0},\mathrm{\infty }),\\ v_t-\mathrm{\Delta }v=v(c-\frac{dv}{u+k}),\begin{array}{cc}& \end{array}\begin{array}{cc}& \end{array}(x,t)\in \mathrm{\Omega }\times (\mathrm{0},\mathrm{\infty }),\\ u=v=\mathrm{0},\begin{array}{cc}& \end{array}x\in \partial \mathrm{\Omega },t\in (\mathrm{0},\mathrm{\infty }),\\ u(x,\mathrm{0})=u_{\mathrm{0}}(x)\ge \mathrm{0},\cancel{\equiv }\mathrm{0},v(x,\mathrm{0})=v_{\mathrm{0}}(x)\ge \mathrm{0},\cancel{\equiv }\mathrm{0},\begin{array}{c}\end{array}x\in \overline{\mathrm{\Omega }}.\end{array}$(2)

Here $u$ and $v$ are the population densities of prey and predator, $a$ and $c$ are the intrinsic growth rates of prey and predator, respectively. The Allee effect constants $m$ and $b$ satisfy $m<ab$ which is called weak additive Allee effect. $dv/(u+k)$ is a modified Leslie-Gower term^[12-13]. The parameter $d$ represents the maximum average reduction rate obtained by predators, and $k$ represents the environmental carrying capacity of predators. For more detailed biological significance of the model, one can see Refs. [14-16]. Obviously, the corresponding steady-state problem to (2) can be written as

$\{\begin{array}{l}-\mathrm{\Delta }u=u(a-u)-\frac{mu}{u+b}-(\mathrm{1}-{\mathrm{e}}^{-\gamma u})v,x\in \mathrm{\Omega },\\ -\mathrm{\Delta }v=v(c-\frac{dv}{u+k}),\begin{array}{cc}& \end{array}x\in \mathrm{\Omega },\\ u=v=\mathrm{0},\begin{array}{cc}& \end{array}\begin{array}{cc}& \end{array}x\in \partial \mathrm{\Omega }.\end{array}$(3)

The main purpose of this paper is to clarify the influence of weak additive Allee effect and Ivlev functional response on the positive solution of model (2). The content of this paper is arranged as follows. Section 1 introduces some preliminary results. Section 2 gives the necessary conditions and prior estimates for positive solutions of (3). Section 3 gives the sufficient conditions for the existence of positive solutions of (3). Section 4 gives the uniqueness and stability of the positive solution of (3). In Section 5, the dynamics of (2) and (3) are quantitatively analyzed by numerical simulations.

1 Preliminaries

Let $q(x)\in C^{\mathrm{1}}(\overline{\mathrm{\Omega }})$. It is well known that the following problem

$\{\begin{array}{l}-\mathrm{\Delta }\varphi +q(x)\varphi =\lambda \varphi ,x\in \mathrm{\Omega },\\ \varphi =\mathrm{0},\begin{array}{cccc}& & & \end{array}x\in \partial \mathrm{\Omega }\end{array}$(4)

has an infinite sequence of eigenvalues which are bounded below. Throughout this paper, we denote the first eigenvalue by ${\lambda }_{\mathrm{1}}(q)$ and the corresponding eigenfunction does not change sign on $\mathrm{\Omega }$. We also denote that ${\lambda }_{\mathrm{1}}={\lambda }_{\mathrm{1}}(\mathrm{0})$ with the corresponding eigenfunction ${\Phi }_{\mathrm{1}}>\mathrm{0},x\in \mathrm{\Omega }$. For more detailed information, one can see Refs. [17-19].

Now we consider the following boundary value problem

$\{\begin{array}{l}-\mathrm{\Delta }u+q(x)u=au-u^{\mathrm{2}},x\in \mathrm{\Omega },\\ u=\mathrm{0},x\in \partial \mathrm{\Omega }.\end{array}$

According to Ref. [20], if $a\le {\lambda }_{\mathrm{1}}(q)$, then u=0 is the unique non-negative solution of this problem, and it has a unique positive solution if $a>{\lambda }_{\mathrm{1}}(q)$. In particular, if $q(x)\equiv \mathrm{0}$ and $a>{\lambda }_{\mathrm{1}}$, then it has a unique positive solution, denoted by ${\theta }_{[a]}$, which is monotonically increasing with respect to $a$. And then, for the boundary value problem

$\{\begin{array}{l}-\mathrm{\Delta }u=u(a-u)-\frac{mu}{u+b},x\in \mathrm{\Omega },\\ u=\mathrm{0},x\in \partial \mathrm{\Omega },\end{array}$(5)

using the upper and lower solution method, it has a unique positive solution $\tilde{u}$ which satisfies $\tilde{u}\le {\theta }_{[a]}\le a$ when $m<b^{\mathrm{2}}$ and $a>{\lambda }_{\mathrm{1}}+m∕b$. We remark here that the condition $a>{\lambda }_{\mathrm{1}}+m/b$ can meet the requirements of the condition of weak additive Allee effect, i.e.,$m<ab$. Finally, consider the following boundary value problem

$\{\begin{array}{l}-\mathrm{\Delta }v=v(c-hv),x\in \mathrm{\Omega },\\ v=\mathrm{0},x\in \partial \mathrm{\Omega }.\end{array}$(6)

It has a unique positive solution, denoted by ${\theta }_{[c,h]}$, if $c>{\lambda }_{\mathrm{1}}$. Moreover, ${\theta }_{[c,h]}\le c∕h$ and is monotonically decreasing with respect to $h$. Especially, when $h=k∕d$, denote the unique positive solution by $\tilde{v}$, that is $\tilde{v}={\theta }_{[c,d/k]}$.

According to (5) and (6), (3) has a unique semi-trivial solution $(\tilde{u},\mathrm{0})$ if $m<b^{\mathrm{2}}$ and $a>{\lambda }_{\mathrm{1}}+m/b$, and has a unique semi-trivial solution $(\mathrm{0},\tilde{v})=(\mathrm{0},{\theta }_{[c,d/k]})$ and ${\theta }_{[c,d/k]}\le ck∕d$ if $c>{\lambda }_{\mathrm{1}}$.

Lemma 1^[21-22] Let $q(x)\in C(\overline{\mathrm{\Omega }})$,$M-q(x)>\mathrm{0}$ with the constant $M$, ${\lambda }_{\mathrm{1}}(q(x))$ be the principal eigenvalue of (4). We have the following statements:

$\begin{array}{l}(\mathrm{a})r[{(M-\mathrm{\Delta })}^{-\mathrm{1}}(M-q(x))]>\mathrm{1}\mathrm{i}\mathrm{f}{\lambda }_{\mathrm{1}}(q(x))<\mathrm{0},\\ (\mathrm{b})r[{(M-\mathrm{\Delta })}^{-\mathrm{1}}(M-q(x))]<\mathrm{1}\mathrm{i}\mathrm{f}{\lambda }_{\mathrm{1}}(q(x))>\mathrm{0},\\ (\mathrm{c})r[{(M-\mathrm{\Delta })}^{-\mathrm{1}}(M-q(x))]=\mathrm{1}\mathrm{i}\mathrm{f}{\lambda }_{\mathrm{1}}(q(x))=\mathrm{0}.\end{array}$

Lemma 2^[22] Let $u(x),v(x)\in C^{\mathrm{1}}(\overline{\mathrm{\Omega }})$ satisfy

$u(x)>\mathrm{0},x\in \mathrm{\Omega },u{|}_{\partial \mathrm{\Omega }}=\mathrm{0};v{|}_{\partial \mathrm{\Omega }}>\mathrm{0},\frac{\partial u}{\partial n}{|}_{\partial \mathrm{\Omega }}<\mathrm{0}.$

Then there exists $\epsilon >\mathrm{0}$ such that $u+\epsilon v>\mathrm{0}$ for any $x\in \mathrm{\Omega }$.

2 Necessary Conditions and Prior Estimates

In this section, we use the upper and lower solution method and the strong maximum principle to establish the necessary condition and a priori estimate of positive solutions of (3).

Theorem 1   If (3) has a positive solution, then $a>{\lambda }_{\mathrm{1}},c>{\lambda }_{\mathrm{1}}$.

Proof   Let $(u,v)$ be a positive solution of (3). Multiply both sides of the first equation of (3) by ${\Phi }_{\mathrm{1}}$ and integrate on $\mathrm{\Omega }$, we can get

${\int }_{\Omega }(a-{\lambda }_{\mathrm{1}}){\Phi }_{\mathrm{1}}u\mathrm{d}x={\int }_{\Omega }(u+\frac{m}{u+b}+\frac{(\mathrm{1}-{\mathrm{e}}^{-\gamma u})v}{u}){\Phi }_{\mathrm{1}}u\mathrm{d}x>\mathrm{0}.$

This implies that $a>{\lambda }_{\mathrm{1}}$. The inequality $c>{\lambda }_{\mathrm{1}}$ can be obtained by the second equation of (3) similarly.

Remark 1   Theorem 1 shows that when the growth rate of predator or prey is low, at least one species is extinct in (3).

Theorem 2   Suppose that $m<b^{\mathrm{2}}$,$a>{\lambda }_{\mathrm{1}}+m/b$ and $(u,v)$ is a positive solution of (3). Then

$\mathrm{0}<u\le \tilde{u}\le {\theta }_{[a]}\le a,\tilde{v}\le v\le {\theta }_{[c,d/(a+k)]}\le \frac{c(a+k)}{d}.$

Proof   The above two inequalities are proved in the same way, and we only prove the second inequality. According to the second equation of (3) we have

$\begin{array}{l}-\mathrm{\Delta }v=v(c-\frac{dv}{u+k})\ge v(c-\frac{dv}{k})\\ -\mathrm{\Delta }v=v(c-\frac{dv}{u+k})\le v(c-\frac{dv}{a+k})\end{array}$

If $(u,v)$ is a positive solution of (3), according to Theorem 1 we have $c>{\lambda }_{\mathrm{1}}$. Then

$-\mathrm{\Delta }v=v(c-\frac{dv}{k}),v{|}_{\partial \mathrm{\Omega }}=\mathrm{0}$and $-\mathrm{\Delta }v=v(c-\frac{dv}{a+k}),v{|}_{\partial \mathrm{\Omega }}=\mathrm{0}$

have a unique positive solution $\tilde{v},{\theta }_{[c,d/(a+k)]}$, respectively. Thus $\tilde{v}\le v\le {\theta }_{[c,d/(a+k)]}$ can be obtained from upper-lower solution method and uniqueness of ${\theta }_{[c,d/(a+k)]}$. The inequality ${\theta }_{[c,d/(a+k)]}\le c(a+k)∕d$ can be obtained from the nature of ${\theta }_{[c,d/(a+k)]}$.

3 Existence of Positive Steady-State Solutions

In this section, we establish the existence of positive solution of (3) by using the degree theory. In order to apply the degree theory, we make the following definitions:

$\begin{array}{l}X=C_{\mathrm{0}}^{\mathrm{1}}(\overline{\mathrm{\Omega }})=:\{u\in C^{\mathrm{1}}(\overline{\mathrm{\Omega }}):u=\mathrm{0},x\in \partial \mathrm{\Omega }\},\\ K=\{u\in C_{\mathrm{0}}^{\mathrm{1}}(\overline{\mathrm{\Omega }}):u\ge \mathrm{0},x\in \overline{\mathrm{\Omega }}\},\\ E=X\times X,W=K\times K,\\ D=\{(u,v)\in W:u<a+\mathrm{1},v<\frac{c(a+k)}{d}+\mathrm{1}\}.\end{array}$

Using Lemma 2, we can get

$\begin{array}{l}\mathrm{1}){\overline{W}}_{(\mathrm{0,0})}=K\times K,S_{(\mathrm{0,0})}=\{(\mathrm{0,0})\},\\ \mathrm{2}){\overline{W}}_{(\tilde{u},\mathrm{0})}=X\times K,S_{(\tilde{u},\mathrm{0})}=X\times \{\mathrm{0}\},\\ \mathrm{3}){\overline{W}}_{(\mathrm{0},\tilde{v})}=K\times X,S_{(\mathrm{0},\tilde{v})}=\{\mathrm{0}\}\times X.\end{array}$

Lemma 3^[22] For the mapping ${\varphi }_t(x):\overline{\mathrm{\Omega }}\times [\mathrm{0,1}]\to {\mathbb{R}}^n$. Suppose that ${\varphi }_t(x)$ is continuous on $\overline{\mathrm{\Omega }}\times [\mathrm{0,1}]$ and ${\varphi }_t(x)\in C^{\mathrm{1}}(\mathrm{\Omega })$ for any $t\in [\mathrm{0,1}]$. If $y_{\mathrm{0}}\cancel{\in }{\varphi }_t(\partial \mathrm{\Omega })$ for any $t\in [\mathrm{0,1}]$, then the topological degree $\mathrm{d}\mathrm{e}\mathrm{g}({\varphi }_t,\mathrm{\Omega },y_{\mathrm{0}})$ does not depend on $t$.

Lemma 4^[22] Let $I-F^{\text{'}}(y)$ be invertible on ${\overline{W}}_y$.

(i) If $F^{\text{'}}(y)$ has $\alpha$-property, then $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,y)=\mathrm{0}$;

(ii) If $F^{\text{'}}(y)$ has no $\alpha$-property, then $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,y)={(-\mathrm{1})}^{\sigma }$, where $\sigma$ is the sum of the algebraic multiplicities of the eigenvalues of $F^{\text{'}}(y)$ which are larger than one.

Lemma 5   Let $m<b^{\mathrm{2}},a>{\lambda }_{\mathrm{1}}+m∕b$. Then all eigenvalues of $J_{\mathrm{0}}$ are greater than 0. Here $J_{\mathrm{0}}$ is the linearization operator of (5) at $\tilde{u}$, i.e.,

$J_{\mathrm{0}}=-\mathrm{\Delta }-(a-\mathrm{2}\tilde{u}-\frac{mb}{{(\tilde{u}+b)}^{\mathrm{2}}}).$

Proof   By $m<b^{\mathrm{2}},a>{\lambda }_{\mathrm{1}}+m∕b$, we have $\tilde{u}$ is a unique positive solution of (5). Thus

${\lambda }_{\mathrm{1}}(-a+\tilde{u}+\frac{m}{\tilde{u}+b})=\mathrm{0}.$

According to $m<b^{\mathrm{2}}$, we have

$\tilde{u}>\frac{m\tilde{u}}{{(\tilde{u}+b)}^{\mathrm{2}}}=\frac{m}{\tilde{u}+b}-\frac{mb}{{(\tilde{u}+b)}^{\mathrm{2}}},$

It means that $\tilde{u}+mb∕{(\tilde{u}+b)}^{\mathrm{2}}>m∕(\tilde{u}+b)$. By the nature of principle eigenvalue, there holds

${\lambda }_{\mathrm{1}}(-a+\mathrm{2}\tilde{u}+\frac{mb}{{(\tilde{u}+b)}^{\mathrm{2}}})>{\lambda }_{\mathrm{1}}(-a+\tilde{u}+\frac{m}{\tilde{u}+b})=\mathrm{0}.$

The proof is completed.

According to Theorem 2, we have any nonnegative solution of (3) belongs to $D$. Then there exists a positive constant $M$, such that

$\begin{array}{l}u(a-u-\frac{m}{u+b}-\frac{(\mathrm{1}-{\mathrm{e}}^{-\gamma u})v}{u})+Mu\ge \mathrm{0},\\ v(c-\frac{dv}{u+k})+Mv\ge \mathrm{0},(u,v)\in \overline{D}\end{array}$

Define mapping $F:E\to E$ as

$F(u,v)={(-\mathrm{\Delta }+M)}^{-\mathrm{1}}\left(\begin{array}{c}u(a-u-\frac{m}{u+b}-\frac{(\mathrm{1}-{\mathrm{e}}^{-\gamma u})v}{u})+Mu\\ v(c-\frac{dv}{u+k})+Mv\end{array}\right).$(7)

Then it is a compact operator and $F:D\to W$. Thus $F(u,v)=(u,v)$ if and only if $(u,v)$ is a solution of (3).

For any $t\in [\mathrm{0,1}]$, we also define

$F_t(u,v)={(-\mathrm{\Delta }+M)}^{-\mathrm{1}}\left(\begin{array}{c}tu(a-u-\frac{m}{u+b}-\frac{(\mathrm{1}-{\mathrm{e}}^{-\gamma u})v}{u})+Mu\\ tv(c-\frac{dv}{u+k})+Mv\end{array}\right).$

Clearly, $F_t(u,v):[\mathrm{0},t]\times D\to W$ is a positive compact operator and $F_{\mathrm{1}}=F$.

Lemma 6   Let $m<b^{\mathrm{2}},a>{\lambda }_{\mathrm{1}}+m∕b$. We have the following statements:

(i) $\mathrm{d}\mathrm{e}{\mathrm{g}}_W(I-F,D)=\mathrm{1},$

(ii) If $c\ne {\lambda }_{\mathrm{1}}$, then $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\mathrm{0,0}))=\mathrm{0}$,

(iii) If $c>{\lambda }_{\mathrm{1}}$, then $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\tilde{u},\mathrm{0}))=\mathrm{0}$,

(iv) If $c<{\lambda }_{\mathrm{1}}$, then $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\tilde{u},\mathrm{0}))=\mathrm{1}$.

Proof   (i) It is clear that $F$ has no fixed point on $\partial \mathrm{\Omega }$. For any $t\in [\mathrm{0,1}]$, the fixed point of $F_t$ is equivalent to the solution of boundary value problem

$\{\begin{array}{l}-\mathrm{\Delta }u=tu(a-u-\frac{m}{u+b}-\frac{(\mathrm{1}-{\mathrm{e}}^{-\gamma u})v}{u}),x\in \mathrm{\Omega },\\ -\mathrm{\Delta }v=tv(c-\frac{dv}{u+k}),x\in \mathrm{\Omega },\\ u=v=\mathrm{0},x\in \partial \mathrm{\Omega }.\end{array}$

According to Theorem 2, the fixed point of $F_t$ satisfies $u\le a,v\le c(a+k)∕d$ for any $t\in [\mathrm{0,1}]$. Thus $\mathrm{d}\mathrm{e}{\mathrm{g}}_W(I-F_t,D)$ does not depend on $t$ from homotopic invariant property and

$\begin{array}{l}\mathrm{d}\mathrm{e}{\mathrm{g}}_W(I-F,D)=\mathrm{d}\mathrm{e}{\mathrm{g}}_W(I-F_{\mathrm{1}},D)\\ =\mathrm{d}\mathrm{e}{\mathrm{g}}_W(I-F_{\mathrm{0}},D)\end{array}$

By the above boundary value problem has a unique solution $(\mathrm{0,0})$ as $t=\mathrm{0}$, we have

$\mathrm{d}\mathrm{e}{\mathrm{g}}_W(I-F_{\mathrm{0}},D)=\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F_{\mathrm{0}},(\mathrm{0,0})).$

Notice ${\overline{W}}_{(\mathrm{0,0})}\setminus S_{(\mathrm{0,0})}=\{K\times K\}\setminus \{(\mathrm{0,0})\}$. Denote $L_{\mathrm{1}}=F_{\mathrm{0}}^{\text{'}}(\mathrm{0,0})$. Then

$L_{\mathrm{1}}={(-\mathrm{\Delta }+M)}^{-\mathrm{1}}\left(\begin{array}{cc}M& \mathrm{0}\\ \mathrm{0}& M\end{array}\right).$

By ${\lambda }_{\mathrm{1}}(\mathrm{0})={\lambda }_{\mathrm{1}}>\mathrm{0}$ and Lemma 1, $r(L_{\mathrm{1}})<\mathrm{1}$. It can derive that $I-L_{\mathrm{1}}$ is invertible on ${\overline{W}}_{(\mathrm{0,0})}$ and $L_{\mathrm{1}}$ has no $\alpha$-property on ${\overline{W}}_{(\mathrm{0,0})}$. According to Lemma 4, we have

$\mathrm{d}\mathrm{e}{\mathrm{g}}_W(I-F,D)=\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F_{\mathrm{0}},(\mathrm{0,0}))=\mathrm{1}.$

(ii) Let $L_{\mathrm{2}}=F^{\text{'}}(\mathrm{0,0})$. Then

$L_{\mathrm{2}}={(-\mathrm{\Delta }+M)}^{-\mathrm{1}}\left(\begin{array}{cc}a-\frac{m}{b}+M& \mathrm{0}\\ \mathrm{0}& c+M\end{array}\right).$

At first, we will show that $I-L_{\mathrm{2}}$ is invertible on ${\overline{W}}_{(\mathrm{0,0})}=K\times K$. If it is not true, then there is $(\xi ,\eta )\in {\overline{W}}_{(\mathrm{0,0})}$ and $(\xi ,\eta )\ne (\mathrm{0,0})$ such that $L_{\mathrm{2}}{(\xi ,\eta )}^{\mathrm{T}}={(\xi ,\eta )}^{\mathrm{T}}$, i.e.,

$\{\begin{array}{l}-\mathrm{\Delta }\xi =(a-\frac{m}{b})\xi ,x\in \mathrm{\Omega },\\ \xi =\mathrm{0},x\in \partial \mathrm{\Omega }.\end{array}$

If $\xi >\mathrm{0}$, then $a-m∕b={\lambda }_{\mathrm{1}}$, this is contrary to $a>m∕b+{\lambda }_{\mathrm{1}}$. Thus $\xi \equiv \mathrm{0}$. Similarly,$\eta \equiv \mathrm{0}$. This is a contradiction with $(\xi ,\eta )\ne (\mathrm{0,0})$. Then $I-L_{\mathrm{2}}$ is invertible on ${\overline{W}}_{(\mathrm{0,0})}$.

Now we claim that $L_{\mathrm{2}}$ has $\alpha$-property on ${\overline{W}}_{(\mathrm{0,0})}$. By $a>m∕b+{\lambda }_{\mathrm{1}}$ and Lemma 1,

$\begin{array}{l}{r}_{\mathrm{1}}=r[{(-\mathrm{\Delta }+M)}^{-\mathrm{1}}(a-\frac{m}{b}+M)]\\ =r[{(-\mathrm{\Delta }+M)}^{-\mathrm{1}}(M-(-a+\frac{m}{b}))]>\mathrm{1}.\end{array}$

Notice that $r_{\mathrm{1}}$ is the principal eigenvalue of the operator ${(-\mathrm{\Delta }+M)}^{-\mathrm{1}}(a-m∕b+M)$, and the corresponding eigenfunction is ${\varphi }_{\mathrm{1}}>\mathrm{0},x\in \mathrm{\Omega }$. Take $t_{\mathrm{1}}=\mathrm{1}∕r_{\mathrm{1}}$, then $\mathrm{0}<t_{\mathrm{1}}<\mathrm{1}$ and $({\varphi }_{\mathrm{1}},\mathrm{0})\in {\overline{W}}_{(\mathrm{0,0})}\setminus S_{(\mathrm{0,0})}$. Thus we have $\begin{array}{l}(I-t_{\mathrm{1}}{L}_{\mathrm{2}})({\varphi }_{\mathrm{1}}{,\mathrm{0})}^{\mathrm{T}}=\left(\begin{array}{c}{\varphi }_{\mathrm{1}}-t_{\mathrm{1}}{(-\mathrm{\Delta }+M)}^{-\mathrm{1}}(a-\frac{m}{b}+M){\varphi }_{\mathrm{1}}\\ \mathrm{0}\end{array}\right)\\ =\left(\begin{array}{c}\mathrm{0}\\ \mathrm{0}\end{array}\right)\in S_{(\mathrm{0,0})}.\end{array}$

Therefore, $L_{\mathrm{2}}$ has $\alpha$-property. By Lemma 4, we have $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\mathrm{0,0}))=\mathrm{0}$.

(iii) Obviously, (3) has a semi-trivial solution $(\tilde{u},\mathrm{0})$. Notice that ${\overline{W}}_{(\tilde{u},\mathrm{0})}=X\times K$, $S_{(\tilde{u},\mathrm{0})}=X\times \{\mathrm{0}\}$, we have ${\overline{W}}_{(\tilde{u},\mathrm{0})}\setminus S_{(\tilde{u},\mathrm{0})}=X\times \{K\setminus \{\mathrm{0}\}\}$. Let $L_{\mathrm{3}}=F^{\text{'}}(\tilde{u},\mathrm{0})$. There holds

$L_{\mathrm{3}}={(-\mathrm{\Delta }+M)}^{-\mathrm{1}}\left(\begin{array}{cc}a-\mathrm{2}\tilde{u}-\frac{mb}{{(\tilde{u}+b)}^{\mathrm{2}}}+M& -(\mathrm{1}-{\mathrm{e}}^{-\gamma \tilde{u}})\eta \\ \mathrm{0}& c+M\end{array}\right).$

Assume that there exists $(\xi ,\eta )\in {\overline{W}}_{(\tilde{u},\mathrm{0})}$ and $(\xi ,\eta )\ne (\mathrm{0,0})$ such that $L_{\mathrm{3}}{(\xi ,\eta )}^{\mathrm{T}}={(\xi ,\eta )}^{\mathrm{T}}$. Then

$\{\begin{array}{l}-\mathrm{\Delta }\xi =(a-\mathrm{2}\tilde{u}-\frac{mb}{{(\tilde{u}+b)}^{\mathrm{2}}})\xi -(\mathrm{1}-{\mathrm{e}}^{-\gamma \tilde{u}})\eta ,x\in \mathrm{\Omega },\\ -\mathrm{\Delta }\eta =c\eta ,x\in \mathrm{\Omega },\\ \xi =\eta =\mathrm{0},x\in \partial \mathrm{\Omega }.\end{array}$

By $c>{\lambda }_{\mathrm{1}}$, we have $\eta \equiv \mathrm{0}$ in $K$. If $\xi \cancel{\equiv }\mathrm{0}$, then the above boundary value problem can be written as

$\{\begin{array}{l}-\mathrm{\Delta }\xi =(a-\mathrm{2}\tilde{u}-\frac{mb}{{(\tilde{u}+b)}^{\mathrm{2}}})\xi ,x\in \mathrm{\Omega },\\ \xi =\mathrm{0},x\in \partial \mathrm{\Omega }.\end{array}$

According to $J_{\mathrm{0}}$ is invertible (see Lemma 5), we have $\xi \equiv \mathrm{0}$. This contradiction leads to $I-L_{\mathrm{3}}$ being invertible on ${\overline{W}}_{(\tilde{u},\mathrm{0})}$.

By Lemma 1 and $c>{\lambda }_{\mathrm{1}}$, we also have

$r_{\mathrm{2}}=r[{(-\mathrm{\Delta }+M)}^{-\mathrm{1}}(c+M)]=r[{(-\mathrm{\Delta }+M)}^{-\mathrm{1}}(M-(-c))]>\mathrm{1}.$

Notice that $r_{\mathrm{2}}$ is the principal eigenvalue of ${(-\mathrm{\Delta }+M)}^{-\mathrm{1}}(c+M)$ and the corresponding eigenfunction ${\varphi }_{\mathrm{2}}>\mathrm{0}$. Take $t_{\mathrm{2}}=\mathrm{1}∕r_{\mathrm{2}}$, then $t_{\mathrm{2}}\in [\mathrm{0,1}]$. Thus $(\mathrm{0},{\varphi }_{\mathrm{2}})\in {\overline{W}}_{(\tilde{u},\mathrm{0})}\setminus S_{(\tilde{u},\mathrm{0})}$, and

$(I-t_{\mathrm{2}}{L}_{\mathrm{3}})(\mathrm{0},{\varphi }_{\mathrm{2}}{)}^{\mathrm{T}}=\left(\begin{array}{c}{t}_{\mathrm{2}}{(-\mathrm{\Delta }+M)}^{-\mathrm{1}}(\mathrm{1}-{\mathrm{e}}^{-\gamma \tilde{u}}){\varphi }_{\mathrm{2}}\\ \mathrm{0}\end{array}\right)\in S_{(\tilde{u},\mathrm{0})}.$

It shows $L_{\mathrm{3}}$ has $\alpha$-properties. According to Lemma 4, there holds $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\tilde{u},\mathrm{0}))=\mathrm{0}$.

(iv) It follows from the proof of (iii) that $L_{\mathrm{3}}$ is invertible on ${\overline{W}}_{(\tilde{u},\mathrm{0})}$.

Now we claim $L_{\mathrm{3}}$ has no $\alpha$-properties on ${\overline{W}}_{(\tilde{u},\mathrm{0})}$. If it is not true, then there exist $\mathrm{0}<t<\mathrm{1}$ and $({\varphi }_{\mathrm{3}},{\varphi }_{\mathrm{4}})\in {\overline{W}}_{(\tilde{u},\mathrm{0})}\setminus S_{(\tilde{u},\mathrm{0})}$ such that $(I-t{L}_{\mathrm{3}})({\varphi }_{\mathrm{3}},{\varphi }_{\mathrm{4}}{)}^{\mathrm{T}}\in S_{(\tilde{u},\mathrm{0})}$. Thus

${\varphi }_{\mathrm{4}}-t{(-\mathrm{\Delta }+M)}^{-\mathrm{1}}(c+M){\varphi }_{\mathrm{4}}=\mathrm{0}.$

Notice that ${\varphi }_{\mathrm{4}}\in K\setminus \{\mathrm{0}\}$, then $\mathrm{1}∕t$ is one eigenvalue of the ${(-\mathrm{\Delta }+M)}^{-\mathrm{1}}(c+M)$. On the other hand, by $c<{\lambda }_{\mathrm{1}}$ and Lemma 1 we have $r_{\mathrm{2}}=r[({(-\mathrm{\Delta }+M)}^{-\mathrm{1}})(c+M))]<\mathrm{1}$, which is a contradiction.

According to $L_{\mathrm{3}}$ has no $\alpha$-property on ${\overline{W}}_{(\tilde{u},\mathrm{0})}$ and Lemma 4, we have

$\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\tilde{u},\mathrm{0}))={(-\mathrm{1})}^{\sigma },$

where $\sigma$ is the sum of the algebraic multiplicities of the eigenvalues of $L_{\mathrm{3}}$ which are larger than one.

Assume that $\mathrm{1}∕\mu >\mathrm{1}$ is the eigenvalue of $L_{\mathrm{3}}$, and the corresponding eigenfunction is ${(\xi ,\eta )}^{\mathrm{T}}$. Then $L_{\mathrm{3}}{(\xi ,\eta )}^{\mathrm{T}}=(\mathrm{1}/\mu ){(\xi ,\eta )}^{\mathrm{T}}$. This can be written as

$\{\begin{array}{l}-\mathrm{\Delta }\xi +M\xi =\mu ((a-\mathrm{2}\tilde{u}-\frac{mb}{{(\tilde{u}+b)}^{\mathrm{2}}}+M)\xi -(\mathrm{1}-{\mathrm{e}}^{-\gamma \tilde{u}})\eta ),x\in \mathrm{\Omega },\\ -\mathrm{\Delta }\eta +M\eta =\mu (c+M)\eta ,x\in \mathrm{\Omega },\\ \xi =\eta =\mathrm{0},x\in \partial \mathrm{\Omega }.\end{array}$

If $\eta \cancel{\equiv }\mathrm{0}$, then the inequality

$\mathrm{0}\ge {\lambda }_{\mathrm{1}}(M(\mathrm{1}-\mu )-\mu c)>{\lambda }_{\mathrm{1}}(-c)={\lambda }_{\mathrm{1}}-c$

holds from the second equation of the above boundary value problem. This is a contradiction with $c<{\lambda }_{\mathrm{1}}$. Thus $\eta \equiv \mathrm{0}$. So $\xi \cancel{\equiv }\mathrm{0}$. Then from the first equation of the above boundary value problem we have

$\begin{array}{l}\mathrm{0}\ge {\lambda }_{\mathrm{1}}(M(\mathrm{1}-\mu )-\mu (a-\mathrm{2}\tilde{u}-\frac{mb}{{(\tilde{u}+b)}^{\mathrm{2}}}))>{\lambda }_{\mathrm{1}}(-a+\tilde{u}+\frac{m}{\tilde{u}+b})=\mathrm{0}\\ \end{array}$

This is an obvious contradiction. Therefore, $L_{\mathrm{3}}$ has no eigenvalue greater than one, that is $\sigma =\mathrm{0}$ and then

$\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\tilde{u},\mathrm{0}))=\mathrm{1}.$

It is similar to Lemma 6 that we can get Lemma 7.

Lemma 7   Let $c>{\lambda }_{\mathrm{1}}$.

(i) If $a>{\lambda }_{\mathrm{1}}(\gamma \tilde{v})+m∕b$, then $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\mathrm{0},\tilde{v}))=\mathrm{0}$,

(ii) If $a<{\lambda }_{\mathrm{1}}(\gamma \tilde{v})+m∕b$, then $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\mathrm{0},\tilde{v}))=\mathrm{1}$,

where $\tilde{v}$ and $F$ are given by (6) and (7), respectively.

Theorem 3   If $m<b^{\mathrm{2}}, a>{\lambda }_{\mathrm{1}}(\gamma \tilde{v})+m∕b, c>{\lambda }_{\mathrm{1}}$, then (3) has at least one positive solution.

Proof   From the additivity of the degree, combining Lemma 6 and Lemma 7 we have

$\begin{array}{l}\mathrm{1}=\mathrm{d}\mathrm{e}\mathrm{g}(I-F,D)\\ \ne \mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\mathrm{0,0}))+\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\tilde{u},\mathrm{0}))+\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}{\mathrm{x}}_W(F,(\mathrm{0},\tilde{v}))\\ =\mathrm{0}+\mathrm{0}+\mathrm{0}.\end{array}$

Therefore, (3) has at least one positive solution.

Remark 2   Theorem 3 shows that predator and prey can coexist as long as the Allee effect constant satisfies the appropriate conditions and the growth rates of predator and prey are appropriately large.

4 Uniqueness and Stability

In this section, we use the stability theory of linear operators to discuss uniqueness and stability of positive steady-state solutions. First, the following Lemma 8 is given.

Lemma 8   Let $m<b^{\mathrm{2}},a>{\lambda }_{\mathrm{1}}+m∕b,c>{\lambda }_{\mathrm{1}}$. There exists a constant $\delta >\mathrm{0}$ small enough, such that any positive solution of (3) is nondegenerate and linearly stable when $\gamma <\delta$ (if the positive solution exists).

Proof   Assume that it is not true. For $\{{\gamma }_i{\}}_{i=\mathrm{1}}^{\mathrm{\infty }}$ with ${\gamma }_i\to \mathrm{0}$, there exists a sequence of positive solutions $\{(u_i,v_i{)\}}_{i=\mathrm{1}}^{\mathrm{\infty }}$ of (3), which are degenerate or unstable.

Now we suppose there are ${\mu }_i$ and $({\xi }_i,{\eta }_i)$, which satisfy $\mathrm{R}\mathrm{e}({\mu }_i)\le \mathrm{0}$ and $\Vert {\xi }_i{\Vert }_{L^{\mathrm{2}}}^{\mathrm{2}}+\Vert {\eta }_i{\Vert }_{L^{\mathrm{2}}}^{\mathrm{2}}=\mathrm{1}$, such that

$L_{(u_i,v_i)}({\xi }_i,{\eta }_i{)}^{\mathrm{T}}={\mu }_i({\xi }_i,{\eta }_i{)}^{\mathrm{T}},$

where $L_{(u_i,v_i)}$ is the linearization operator of (3) at $(u_i,v_i)$, i.e.,

$\{\begin{array}{l}\begin{array}{l}-\mathrm{\Delta }{\xi }_i-(a-\mathrm{2}{u}_i){\xi }_i+\frac{mb}{(u_i{+b)}^{\mathrm{2}}}{\xi }_i\\ +{\gamma }_i{v}_i{\mathrm{e}}^{-{\gamma }_i{u}_i}{\xi }_i+(\mathrm{1}-{\mathrm{e}}^{-{\gamma }_i{u}_i}){\eta }_i={\mu }_i{\xi }_i,x\in \mathrm{\Omega },\end{array}\\ -\mathrm{\Delta }{\eta }_i-(c-\frac{\mathrm{2}d{v}_i}{u_i+k}){\eta }_i-\frac{d{v}_i^{\mathrm{2}}}{(u_i{+k)}^{\mathrm{2}}}{\xi }_i={\mu }_i{\eta }_i,x\in \mathrm{\Omega },\\ {\xi }_i={\eta }_i=\mathrm{0},x\in \partial \mathrm{\Omega }.\end{array}$(8)