\chapter{The distributed-\/solver program }
\hypertarget{distributed_solver}{}\label{distributed_solver}\index{The distributed-\/solver program@{The distributed-\/solver program}}
The distributed solver example.

This example depends on simple-\/solver, three-\/pt-\/stencil-\/solver.

 \label{three_pt_stencil_solver_Intro}%
\Hypertarget{three_pt_stencil_solver_Intro}%
 \label{three_pt_stencil_solver_Introduction}%
\Hypertarget{three_pt_stencil_solver_Introduction}%
\doxysection*{Introduction}

This distributed solver example should help you understand the basics of using Ginkgo in a distributed setting. The example will solve a simple 1D Laplace equation where the system can be distributed row-\/wise to multiple processes. To run the solver with multiple processes, use {\ttfamily mpirun -\/n NUM\+\_\+\+PROCS ./\+distributed-\/solver \+[executor]\+ \+[num\+\_\+grid\+\_\+points]\+ \+[num\+\_\+iterations]\+}.

If you are using GPU devices, please make sure that you run this example with at most as many processes as you have GPU devices available. \label{three_pt_stencil_solver_CommProg}%
\Hypertarget{three_pt_stencil_solver_CommProg}%
 \doxysection*{The commented program}

As vector type we use the following, which implements a subset of \doxylink{classgko_1_1matrix_1_1Dense}{gko::\+matrix::\+\+Dense}.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{using\ }dist\_vec\ =\ \doxymbox{\hyperlink{classgko_1_1experimental_1_1distributed_1_1Vector}{gko::experimental::distributed::Vector<ValueType>}};}

\end{DoxyCode}


As matrix type we simply use the following type, which can read distributed data and be applied to a distributed vector.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{using\ }dist\_mtx\ =}
\DoxyCodeLine{\ \ \ \ \doxymbox{\hyperlink{classgko_1_1experimental_1_1distributed_1_1Matrix}{gko::experimental::distributed::Matrix}}<ValueType,\ LocalIndexType,}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ GlobalIndexType>;}

\end{DoxyCode}


We still need a localized vector type to be used as scalars in the advanced apply operations.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{using\ }vec\ =\ \doxymbox{\hyperlink{classgko_1_1matrix_1_1Dense}{gko::matrix::Dense<ValueType>}};}

\end{DoxyCode}


The partition type describes how the rows of the matrices are distributed.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{using\ }part\_type\ =}
\DoxyCodeLine{\ \ \ \ \doxymbox{\hyperlink{classgko_1_1experimental_1_1distributed_1_1Partition}{gko::experimental::distributed::Partition}}<LocalIndexType,}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ GlobalIndexType>;}

\end{DoxyCode}


We can use here the same solver type as you would use in a non-\/distributed program. Please note that not all solvers support distributed systems at the moment.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{using\ }\doxymbox{\hyperlink{namespacegko_1_1log_a15f71654c6c4f4defc95f9359343fa27a7cbf673200498a0d1324f4330825202a}{solver}}\ =\ \doxymbox{\hyperlink{classgko_1_1solver_1_1Cg}{gko::solver::Cg<ValueType>}};}
\DoxyCodeLine{\textcolor{keyword}{using\ }schwarz\ =\ \doxymbox{\hyperlink{classgko_1_1experimental_1_1distributed_1_1preconditioner_1_1Schwarz}{gko::experimental::distributed::preconditioner::Schwarz}}<}
\DoxyCodeLine{\ \ \ \ ValueType,\ LocalIndexType,\ GlobalIndexType>;}
\DoxyCodeLine{\textcolor{keyword}{using\ }bj\ =\ \doxymbox{\hyperlink{classgko_1_1preconditioner_1_1Jacobi}{gko::preconditioner::Jacobi<ValueType,\ LocalIndexType>}};}
\DoxyCodeLine{\textcolor{keyword}{using\ }pgm\ =\ \doxymbox{\hyperlink{classgko_1_1multigrid_1_1Pgm}{gko::multigrid::Pgm<ValueType,\ LocalIndexType>}};}

\end{DoxyCode}


Create an MPI communicator get the rank of the calling process.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ comm\ =\ \doxymbox{\hyperlink{classgko_1_1experimental_1_1mpi_1_1communicator}{gko::experimental::mpi::communicator}}(MPI\_COMM\_WORLD);}
\DoxyCodeLine{\textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ rank\ =\ comm.rank();}

\end{DoxyCode}


\label{distributed_solver_UserInputHandling}%
\Hypertarget{distributed_solver_UserInputHandling}%
 \doxysubsubsection*{User Input Handling}

User input settings:\+
\begin{DoxyItemize}
\item The executor, defaults to reference.
\item The number of grid points, defaults to 100.
\item The number of iterations, defaults to 1000.
\item One-\/level, two-\/level preconditioner, and no preconditioner, defaults to one-\/level.
\end{DoxyItemize}


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keywordflow}{if}\ (argc\ ==\ 2\ \&\&\ (std::string(argv[1])\ ==\ \textcolor{stringliteral}{"{}-\/-\/help"{}}))\ \{}
\DoxyCodeLine{\ \ \ \ \textcolor{keywordflow}{if}\ (rank\ ==\ 0)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ std::cerr\ <<\ \textcolor{stringliteral}{"{}Usage:\ "{}}\ <<\ argv[0]}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\ [executor]\ [num\_grid\_points]\ [num\_iterations]\ "{}}}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{stringliteral}{"{}[schwarz\_prec\_type]\ "{}}}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ std::endl;}
\DoxyCodeLine{\ \ \ \ \}}
\DoxyCodeLine{\ \ \ \ std::exit(-\/1);}
\DoxyCodeLine{\}}
\DoxyCodeLine{}
\DoxyCodeLine{ValueType\ t\_init\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a508f97387781bb32dd2c7616dee1a786}{gko::experimental::mpi::get\_walltime}}();}
\DoxyCodeLine{}
\DoxyCodeLine{\textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ executor\_string\ =\ argc\ >=\ 2\ ?\ argv[1]\ :\ \textcolor{stringliteral}{"{}reference"{}};}
\DoxyCodeLine{\textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ grid\_dim\ =}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{static\_cast<}\doxymbox{\hyperlink{namespacegko_a9579c98b1d85bf4ad02d7cb5b96fc771}{gko::size\_type}}\textcolor{keyword}{>}(argc\ >=\ 3\ ?\ std::atoi(argv[2])\ :\ 100);}
\DoxyCodeLine{\textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ num\_iters\ =}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{static\_cast<}\doxymbox{\hyperlink{namespacegko_a9579c98b1d85bf4ad02d7cb5b96fc771}{gko::size\_type}}\textcolor{keyword}{>}(argc\ >=\ 4\ ?\ std::atoi(argv[3])\ :\ 1000);}
\DoxyCodeLine{std::string\ schw\_type\ =\ argc\ >=\ 5\ ?\ argv[4]\ :\ \textcolor{stringliteral}{"{}one-\/level"{}};}
\DoxyCodeLine{}
\DoxyCodeLine{\textcolor{keyword}{const}\ std::map<std::string,}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ std::function<std::shared\_ptr<gko::Executor>(MPI\_Comm)>>}
\DoxyCodeLine{\ \ \ \ executor\_factory\_mpi\{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \{\textcolor{stringliteral}{"{}reference"{}},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ [](MPI\_Comm)\ \{\ \textcolor{keywordflow}{return}\ gko::ReferenceExecutor::create();\ \}\},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \{\textcolor{stringliteral}{"{}omp"{}},\ [](MPI\_Comm)\ \{\ \textcolor{keywordflow}{return}\ \doxymbox{\hyperlink{classgko_1_1OmpExecutor_af977cb680e0427ca4eb71eb7b1266211}{gko::OmpExecutor::create}}();\ \}\},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \{\textcolor{stringliteral}{"{}cuda"{}},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ [](MPI\_Comm\ comm)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordtype}{int}\ device\_id\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a0ec0c93a068b2c19da8cb5e874bf9f6c}{gko::experimental::mpi::map\_rank\_to\_device\_id}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ comm,\ \doxymbox{\hyperlink{classgko_1_1CudaExecutor_aef0258494d14de0e56149b920c5173e5}{gko::CudaExecutor::get\_num\_devices}}());}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordflow}{return}\ \doxymbox{\hyperlink{classgko_1_1CudaExecutor_a075a1957241de65c0cad59f9534430a2}{gko::CudaExecutor::create}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ device\_id,\ gko::ReferenceExecutor::create());}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \}\},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \{\textcolor{stringliteral}{"{}hip"{}},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ [](MPI\_Comm\ comm)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordtype}{int}\ device\_id\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a0ec0c93a068b2c19da8cb5e874bf9f6c}{gko::experimental::mpi::map\_rank\_to\_device\_id}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ comm,\ \doxymbox{\hyperlink{classgko_1_1HipExecutor_a7b34cbd84692dafa2290e6630754aa97}{gko::HipExecutor::get\_num\_devices}}());}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordflow}{return}\ \doxymbox{\hyperlink{classgko_1_1HipExecutor_a8afc5210bf8dd8aedfceab4b73f0047b}{gko::HipExecutor::create}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ device\_id,\ gko::ReferenceExecutor::create());}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \}\},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \{\textcolor{stringliteral}{"{}dpcpp"{}},\ [](MPI\_Comm\ comm)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordtype}{int}\ device\_id\ =\ 0;}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordflow}{if}\ (\doxymbox{\hyperlink{classgko_1_1DpcppExecutor_adf306338233c8e8aca0dec96aefe5d93}{gko::DpcppExecutor::get\_num\_devices}}(\textcolor{stringliteral}{"{}gpu"{}}))\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ device\_id\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a0ec0c93a068b2c19da8cb5e874bf9f6c}{gko::experimental::mpi::map\_rank\_to\_device\_id}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ comm,\ \doxymbox{\hyperlink{classgko_1_1DpcppExecutor_adf306338233c8e8aca0dec96aefe5d93}{gko::DpcppExecutor::get\_num\_devices}}(\textcolor{stringliteral}{"{}gpu"{}}));}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \}\ \textcolor{keywordflow}{else}\ \textcolor{keywordflow}{if}\ (\doxymbox{\hyperlink{classgko_1_1DpcppExecutor_adf306338233c8e8aca0dec96aefe5d93}{gko::DpcppExecutor::get\_num\_devices}}(\textcolor{stringliteral}{"{}cpu"{}}))\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ device\_id\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a0ec0c93a068b2c19da8cb5e874bf9f6c}{gko::experimental::mpi::map\_rank\_to\_device\_id}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ comm,\ \doxymbox{\hyperlink{classgko_1_1DpcppExecutor_adf306338233c8e8aca0dec96aefe5d93}{gko::DpcppExecutor::get\_num\_devices}}(\textcolor{stringliteral}{"{}cpu"{}}));}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \}\ \textcolor{keywordflow}{else}\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordflow}{throw}\ std::runtime\_error(\textcolor{stringliteral}{"{}No\ suitable\ DPC++\ devices"{}});}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \}}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordflow}{return}\ \doxymbox{\hyperlink{classgko_1_1DpcppExecutor_af69a356ee07f641010df5d2e9e518bb0}{gko::DpcppExecutor::create}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ device\_id,\ gko::ReferenceExecutor::create());}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \}\}\};}
\DoxyCodeLine{}
\DoxyCodeLine{\textcolor{keyword}{auto}\ exec\ =\ executor\_factory\_mpi.at(executor\_string)(MPI\_COMM\_WORLD);}

\end{DoxyCode}


\label{distributed_solver_CreatingtheDistributedMatrixandVectors}%
\Hypertarget{distributed_solver_CreatingtheDistributedMatrixandVectors}%
 \doxysubsubsection*{Creating the Distributed Matrix and Vectors}

As a first step, we create a partition of the rows. The partition consists of ranges of consecutive rows which are assigned a part-\/id. These part-\/ids will be used for the distributed data structures to determine which rows will be stored locally. In this example each rank has (nearly) the same number of rows, so we can use the following specialized constructor. See gko::\+distributed::\+\+Partition for other modes of creating a partition.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ num\_rows\ =\ grid\_dim;}
\DoxyCodeLine{\textcolor{keyword}{auto}\ partition\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(part\_type::build\_from\_global\_size\_uniform(}
\DoxyCodeLine{\ \ \ \ exec-\/>get\_master(),\ comm.size(),}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{static\_cast<}GlobalIndexType\textcolor{keyword}{>}(num\_rows)));}

\end{DoxyCode}


Assemble the matrix using a 3-\/pt stencil and fill the right-\/hand-\/side with a sine value. The distributed matrix supports only constructing an empty matrix of zero size and filling in the values with \doxylink{classgko_1_1experimental_1_1distributed_1_1Matrix_aac99e854ec33cee5947041acdf0f1a19}{gko::\+experimental::\+distributed::\+\+Matrix::\+read\+\_\+distributed}. Only the data that belongs to the rows by this rank will be assembled.


\begin{DoxyCode}{0}
\DoxyCodeLine{\doxymbox{\hyperlink{structgko_1_1matrix__data}{gko::matrix\_data<ValueType,\ GlobalIndexType>}}\ A\_data;}
\DoxyCodeLine{\doxymbox{\hyperlink{structgko_1_1matrix__data}{gko::matrix\_data<ValueType,\ GlobalIndexType>}}\ b\_data;}
\DoxyCodeLine{\doxymbox{\hyperlink{structgko_1_1matrix__data}{gko::matrix\_data<ValueType,\ GlobalIndexType>}}\ x\_data;}
\DoxyCodeLine{A\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a30716bd9e39a7beed52fc56d737fc07b}{size}}\ =\ \{num\_rows,\ num\_rows\};}
\DoxyCodeLine{b\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a30716bd9e39a7beed52fc56d737fc07b}{size}}\ =\ \{num\_rows,\ 1\};}
\DoxyCodeLine{x\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a30716bd9e39a7beed52fc56d737fc07b}{size}}\ =\ \{num\_rows,\ 1\};}
\DoxyCodeLine{\textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ range\_start\ =\ partition-\/>get\_range\_bounds()[rank];}
\DoxyCodeLine{\textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ range\_end\ =\ partition-\/>get\_range\_bounds()[rank\ +\ 1];}
\DoxyCodeLine{\textcolor{keywordflow}{for}\ (\textcolor{keywordtype}{int}\ i\ =\ range\_start;\ i\ <\ range\_end;\ i++)\ \{}
\DoxyCodeLine{\ \ \ \ \textcolor{keywordflow}{if}\ (i\ >\ 0)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ A\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a4f6be92270fa96bccdc502c83d248dc0}{nonzeros}}.emplace\_back(i,\ i\ -\/\ 1,\ -\/1);}
\DoxyCodeLine{\ \ \ \ \}}
\DoxyCodeLine{\ \ \ \ A\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a4f6be92270fa96bccdc502c83d248dc0}{nonzeros}}.emplace\_back(i,\ i,\ 2);}
\DoxyCodeLine{\ \ \ \ \textcolor{keywordflow}{if}\ (i\ <\ grid\_dim\ -\/\ 1)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ A\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a4f6be92270fa96bccdc502c83d248dc0}{nonzeros}}.emplace\_back(i,\ i\ +\ 1,\ -\/1);}
\DoxyCodeLine{\ \ \ \ \}}
\DoxyCodeLine{\ \ \ \ b\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a4f6be92270fa96bccdc502c83d248dc0}{nonzeros}}.emplace\_back(i,\ 0,\ std::sin(i\ *\ 0.01));}
\DoxyCodeLine{\ \ \ \ x\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a4f6be92270fa96bccdc502c83d248dc0}{nonzeros}}.emplace\_back(i,\ 0,\ \doxymbox{\hyperlink{namespacegko_a70dbe01ff95c7b953d3d737424c6feb5}{gko::zero<ValueType>}}());}
\DoxyCodeLine{\}}

\end{DoxyCode}


Take timings.


\begin{DoxyCode}{0}
\DoxyCodeLine{comm.synchronize();}
\DoxyCodeLine{ValueType\ t\_init\_end\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a508f97387781bb32dd2c7616dee1a786}{gko::experimental::mpi::get\_walltime}}();}

\end{DoxyCode}


Read the matrix data, currently this is only supported on CPU executors. This will also set up the communication pattern needed for the distributed matrix-\/vector multiplication.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{auto}\ A\_host\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(dist\_mtx::create(exec-\/>get\_master(),\ comm));}
\DoxyCodeLine{\textcolor{keyword}{auto}\ x\_host\ =\ dist\_vec::create(exec-\/>get\_master(),\ comm);}
\DoxyCodeLine{\textcolor{keyword}{auto}\ b\_host\ =\ dist\_vec::create(exec-\/>get\_master(),\ comm);}
\DoxyCodeLine{A\_host-\/>read\_distributed(A\_data,\ partition);}
\DoxyCodeLine{b\_host-\/>read\_distributed(b\_data,\ partition);}
\DoxyCodeLine{x\_host-\/>read\_distributed(x\_data,\ partition);}

\end{DoxyCode}


After reading, the matrix and vector can be moved to the chosen executor, since the distributed matrix supports Sp\+MV also on devices.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{auto}\ A\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(dist\_mtx::create(exec,\ comm));}
\DoxyCodeLine{\textcolor{keyword}{auto}\ x\ =\ dist\_vec::create(exec,\ comm);}
\DoxyCodeLine{\textcolor{keyword}{auto}\ b\ =\ dist\_vec::create(exec,\ comm);}
\DoxyCodeLine{A-\/>copy\_from(A\_host);}
\DoxyCodeLine{b-\/>copy\_from(b\_host);}
\DoxyCodeLine{x-\/>copy\_from(x\_host);}

\end{DoxyCode}


Take timings.


\begin{DoxyCode}{0}
\DoxyCodeLine{comm.synchronize();}
\DoxyCodeLine{ValueType\ t\_read\_setup\_end\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a508f97387781bb32dd2c7616dee1a786}{gko::experimental::mpi::get\_walltime}}();}

\end{DoxyCode}


\label{distributed_solver_SolvetheDistributedSystem}%
\Hypertarget{distributed_solver_SolvetheDistributedSystem}%
 \doxysubsubsection*{Solve the Distributed System}

Generate the solver, this is the same as in the non-\/distributed case. with a local block diagonal preconditioner.

Setup the local block diagonal solver factory.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{auto}\ local\_solver\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(bj::build().on(exec));}

\end{DoxyCode}


Setup the coarse solver. If it is more accurate, then the outer iterations will reduce, but the cost of the coarse solve increases. The coarse solver can in turn have another Schwarz preconditioner if needed.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{auto}\ coarse\_solver\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(}
\DoxyCodeLine{\ \ \ \ solver::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ .with\_preconditioner(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ schwarz::build().with\_local\_solver(local\_solver).on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ .with\_criteria(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ gko::stop::Iteration::build().with\_max\_iters(1000u).on(exec),}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ gko::stop::ResidualNorm<ValueType>::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_reduction\_factor(1e-\/7)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ .on(exec));}
\DoxyCodeLine{}
\DoxyCodeLine{\textcolor{keyword}{auto}\ pgm\_fac\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(pgm::build().on(exec));}

\end{DoxyCode}


Setup the stopping criterion and logger


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{const}\ \doxymbox{\hyperlink{namespacegko_a107302a750bfa82a2107c77293f791c9}{gko::remove\_complex<ValueType>}}\ reduction\_factor\{1e-\/8\};}
\DoxyCodeLine{std::shared\_ptr<const\ gko::log::Convergence<ValueType>>\ logger\ =}
\DoxyCodeLine{\ \ \ \ \doxymbox{\hyperlink{classgko_1_1log_1_1Convergence_ab0ec7b5f1f08c7237e2181e71cf7d5ed}{gko::log::Convergence<ValueType>::create}}();}
\DoxyCodeLine{std::shared\_ptr<gko::LinOp>\ Ainv\{\};}

\end{DoxyCode}


The benefit of the two-\/level Schwarz preconditioner is generally observable (in runtime and in number of iterations) usually for larger problem sizes and for larger number of ranks.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keywordflow}{if}\ (schw\_type\ ==\ \textcolor{stringliteral}{"{}two-\/level"{}})\ \{}
\DoxyCodeLine{\ \ \ \ Ainv\ =}
\DoxyCodeLine{\ \ \ \ \ \ \ \ solver::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ .with\_preconditioner(schwarz::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_local\_solver(local\_solver)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_coarse\_level(pgm\_fac)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_coarse\_solver(coarse\_solver)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ .with\_criteria(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::Iteration::build().with\_max\_iters(num\_iters).on(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ exec),}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::ResidualNorm<ValueType>::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_reduction\_factor(reduction\_factor)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ .on(exec)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ -\/>generate(A);}
\DoxyCodeLine{\}\ \textcolor{keywordflow}{else}\ \textcolor{keywordflow}{if}\ (schw\_type\ ==\ \textcolor{stringliteral}{"{}one-\/level"{}})\ \{}
\DoxyCodeLine{\ \ \ \ Ainv\ =}
\DoxyCodeLine{\ \ \ \ \ \ \ \ solver::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ .with\_preconditioner(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ schwarz::build().with\_local\_solver(local\_solver).on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ .with\_criteria(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::Iteration::build().with\_max\_iters(num\_iters).on(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ exec),}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::ResidualNorm<ValueType>::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_reduction\_factor(reduction\_factor)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ .on(exec)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ -\/>generate(A);}
\DoxyCodeLine{\}\ \textcolor{keywordflow}{else}\ \{}
\DoxyCodeLine{\ \ \ \ schw\_type\ =\ \textcolor{stringliteral}{"{}no-\/precond"{}};}
\DoxyCodeLine{\ \ \ \ Ainv\ =}
\DoxyCodeLine{\ \ \ \ \ \ \ \ solver::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ .with\_criteria(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::Iteration::build().with\_max\_iters(num\_iters).on(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ exec),}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::ResidualNorm<ValueType>::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_reduction\_factor(reduction\_factor)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ .on(exec)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ -\/>generate(A);}
\DoxyCodeLine{\}}

\end{DoxyCode}


Add logger to the generated solver to log the iteration count and residual norm


\begin{DoxyCode}{0}
\DoxyCodeLine{Ainv-\/>add\_logger(logger);}

\end{DoxyCode}


Take timings.


\begin{DoxyCode}{0}
\DoxyCodeLine{comm.synchronize();}
\DoxyCodeLine{ValueType\ t\_solver\_generate\_end\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a508f97387781bb32dd2c7616dee1a786}{gko::experimental::mpi::get\_walltime}}();}

\end{DoxyCode}


Apply the distributed solver, this is the same as in the non-\/distributed case.


\begin{DoxyCode}{0}
\DoxyCodeLine{Ainv-\/>apply(b,\ x);}

\end{DoxyCode}


Take timings.


\begin{DoxyCode}{0}
\DoxyCodeLine{comm.synchronize();}
\DoxyCodeLine{ValueType\ t\_end\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a508f97387781bb32dd2c7616dee1a786}{gko::experimental::mpi::get\_walltime}}();}

\end{DoxyCode}


Get the residual.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keyword}{auto}\ res\_norm\ =\ \doxymbox{\hyperlink{namespacegko_ae3e355a655feea026a7c46814d37a9c9}{gko::as<vec>}}(logger-\/>get\_residual\_norm());}
\DoxyCodeLine{\textcolor{keyword}{auto}\ host\_res\ =\ \doxymbox{\hyperlink{namespacegko_adf2f8602dc9453752a787b40bbf13fcf}{gko::make\_temporary\_clone}}(exec-\/>get\_master(),\ res\_norm);}

\end{DoxyCode}


\label{distributed_solver_PrintingResults}%
\Hypertarget{distributed_solver_PrintingResults}%
 \doxysubsubsection*{Printing Results}

Print the achieved residual norm and timings on rank 0.


\begin{DoxyCode}{0}
\DoxyCodeLine{\textcolor{keywordflow}{if}\ (comm.rank()\ ==\ 0)\ \{}

\end{DoxyCode}


clang-\/format off


\begin{DoxyCode}{0}
\DoxyCodeLine{std::cout\ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nNum\ rows\ in\ matrix:\ "{}}\ <<\ num\_rows}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nNum\ ranks:\ "{}}\ <<\ comm.size()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nPrecond\ type:\ "{}}\ <<\ schw\_type}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nFinal\ Res\ norm:\ "{}}\ <<\ *host\_res-\/>get\_const\_values()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nIteration\ count:\ "{}}\ <<\ logger-\/>get\_num\_iterations()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nInit\ time:\ "{}}\ <<\ t\_init\_end\ -\/\ t\_init}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nRead\ time:\ "{}}\ <<\ t\_read\_setup\_end\ -\/\ t\_init}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nSolver\ generate\ time:\ "{}}\ <<\ t\_solver\_generate\_end\ -\/\ t\_read\_setup\_end}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nSolver\ apply\ time:\ "{}}\ <<\ t\_end\ -\/\ t\_solver\_generate\_end}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nTotal\ time:\ "{}}\ <<\ t\_end\ -\/\ t\_init}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ <<\ std::endl;}

\end{DoxyCode}


clang-\/format on


\begin{DoxyCode}{0}
\DoxyCodeLine{\ \ \ \ \}}
\DoxyCodeLine{\}}

\end{DoxyCode}
 \label{three_pt_stencil_solver_Results}%
\Hypertarget{three_pt_stencil_solver_Results}%
\doxysection*{Results}

This is the expected output for {\ttfamily mpirun -\/n 4 ./\+distributed-\/solver}:\+


\begin{DoxyCode}{0}
\DoxyCodeLine{Num\ \doxymbox{\hyperlink{namespacegko_1_1matrix_ade1f3cdbf05032c2d681fa151de02972adf347a373b8f92aa0ae3dd920a5ec2f6}{rows}}\ in\ matrix:\ 100}
\DoxyCodeLine{Num\ ranks:\ 4}
\DoxyCodeLine{Final\ Res\ norm:\ 5.58392e-\/12}
\DoxyCodeLine{Iteration\ count:\ 7}
\DoxyCodeLine{Init\ time:\ 0.0663887}
\DoxyCodeLine{Read\ time:\ 0.0729806}
\DoxyCodeLine{Solver\ generate\ time:\ 7.6348e-\/05}
\DoxyCodeLine{Solver\ apply\ time:\ 0.0680783}
\DoxyCodeLine{Total\ time:\ 0.141351}

\end{DoxyCode}


The timings may vary depending on the machine. \label{three_pt_stencil_solver_PlainProg}%
\Hypertarget{three_pt_stencil_solver_PlainProg}%
 \doxysection*{The plain program}


\begin{DoxyCodeInclude}{0}
\DoxyCodeLine{}
\DoxyCodeLine{}
\DoxyCodeLine{\textcolor{preprocessor}{\#include\ <ginkgo/ginkgo.hpp>}}
\DoxyCodeLine{}
\DoxyCodeLine{\textcolor{preprocessor}{\#include\ <iostream>}}
\DoxyCodeLine{\textcolor{preprocessor}{\#include\ <map>}}
\DoxyCodeLine{\textcolor{preprocessor}{\#include\ <string>}}
\DoxyCodeLine{}
\DoxyCodeLine{}
\DoxyCodeLine{\textcolor{keywordtype}{int}\ main(\textcolor{keywordtype}{int}\ argc,\ \textcolor{keywordtype}{char}*\ argv[])}
\DoxyCodeLine{\{}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{const}\ \doxymbox{\hyperlink{classgko_1_1experimental_1_1mpi_1_1environment}{gko::experimental::mpi::environment}}\ env(argc,\ argv);}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{using\ }GlobalIndexType\ =\ \doxymbox{\hyperlink{namespacegko_a913ec044df518649fa5a891f8dffae34}{gko::int64}};}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{using\ }LocalIndexType\ =\ \doxymbox{\hyperlink{namespacegko_a16bd14c10900eb804d784908d8aab5ab}{gko::int32}};}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{using\ }ValueType\ =\ double;}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{using\ }dist\_vec\ =\ \doxymbox{\hyperlink{classgko_1_1experimental_1_1distributed_1_1Vector}{gko::experimental::distributed::Vector<ValueType>}};}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{using\ }dist\_mtx\ =}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \doxymbox{\hyperlink{classgko_1_1experimental_1_1distributed_1_1Matrix}{gko::experimental::distributed::Matrix}}<ValueType,\ LocalIndexType,}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ GlobalIndexType>;}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{using\ }vec\ =\ \doxymbox{\hyperlink{classgko_1_1matrix_1_1Dense}{gko::matrix::Dense<ValueType>}};}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{using\ }part\_type\ =}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \doxymbox{\hyperlink{classgko_1_1experimental_1_1distributed_1_1Partition}{gko::experimental::distributed::Partition}}<LocalIndexType,}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ GlobalIndexType>;}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{using\ }\doxymbox{\hyperlink{namespacegko_1_1log_a15f71654c6c4f4defc95f9359343fa27a7cbf673200498a0d1324f4330825202a}{solver}}\ =\ \doxymbox{\hyperlink{classgko_1_1solver_1_1Cg}{gko::solver::Cg<ValueType>}};}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{using\ }schwarz\ =\ \doxymbox{\hyperlink{classgko_1_1experimental_1_1distributed_1_1preconditioner_1_1Schwarz}{gko::experimental::distributed::preconditioner::Schwarz}}<}
\DoxyCodeLine{\ \ \ \ \ \ \ \ ValueType,\ LocalIndexType,\ GlobalIndexType>;}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{using\ }bj\ =\ \doxymbox{\hyperlink{classgko_1_1preconditioner_1_1Jacobi}{gko::preconditioner::Jacobi<ValueType,\ LocalIndexType>}};}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{using\ }pgm\ =\ \doxymbox{\hyperlink{classgko_1_1multigrid_1_1Pgm}{gko::multigrid::Pgm<ValueType,\ LocalIndexType>}};}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ comm\ =\ \doxymbox{\hyperlink{classgko_1_1experimental_1_1mpi_1_1communicator}{gko::experimental::mpi::communicator}}(MPI\_COMM\_WORLD);}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ rank\ =\ comm.rank();}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keywordflow}{if}\ (argc\ ==\ 2\ \&\&\ (std::string(argv[1])\ ==\ \textcolor{stringliteral}{"{}-\/-\/help"{}}))\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \textcolor{keywordflow}{if}\ (rank\ ==\ 0)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ std::cerr\ <<\ \textcolor{stringliteral}{"{}Usage:\ "{}}\ <<\ argv[0]}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\ [executor]\ [num\_grid\_points]\ [num\_iterations]\ "{}}}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{stringliteral}{"{}[schwarz\_prec\_type]\ "{}}}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ std::endl;}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \}}
\DoxyCodeLine{\ \ \ \ \ \ \ \ std::exit(-\/1);}
\DoxyCodeLine{\ \ \ \ \}}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ ValueType\ t\_init\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a508f97387781bb32dd2c7616dee1a786}{gko::experimental::mpi::get\_walltime}}();}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ executor\_string\ =\ argc\ >=\ 2\ ?\ argv[1]\ :\ \textcolor{stringliteral}{"{}reference"{}};}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ grid\_dim\ =}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \textcolor{keyword}{static\_cast<}\doxymbox{\hyperlink{namespacegko_a9579c98b1d85bf4ad02d7cb5b96fc771}{gko::size\_type}}\textcolor{keyword}{>}(argc\ >=\ 3\ ?\ std::atoi(argv[2])\ :\ 100);}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ num\_iters\ =}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \textcolor{keyword}{static\_cast<}\doxymbox{\hyperlink{namespacegko_a9579c98b1d85bf4ad02d7cb5b96fc771}{gko::size\_type}}\textcolor{keyword}{>}(argc\ >=\ 4\ ?\ std::atoi(argv[3])\ :\ 1000);}
\DoxyCodeLine{\ \ \ \ std::string\ schw\_type\ =\ argc\ >=\ 5\ ?\ argv[4]\ :\ \textcolor{stringliteral}{"{}one-\/level"{}};}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{const}\ std::map<std::string,}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ std::function<std::shared\_ptr<gko::Executor>(MPI\_Comm)>>}
\DoxyCodeLine{\ \ \ \ \ \ \ \ executor\_factory\_mpi\{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \{\textcolor{stringliteral}{"{}reference"{}},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ [](MPI\_Comm)\ \{\ \textcolor{keywordflow}{return}\ gko::ReferenceExecutor::create();\ \}\},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \{\textcolor{stringliteral}{"{}omp"{}},\ [](MPI\_Comm)\ \{\ \textcolor{keywordflow}{return}\ \doxymbox{\hyperlink{classgko_1_1OmpExecutor_af977cb680e0427ca4eb71eb7b1266211}{gko::OmpExecutor::create}}();\ \}\},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \{\textcolor{stringliteral}{"{}cuda"{}},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ [](MPI\_Comm\ comm)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordtype}{int}\ device\_id\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a0ec0c93a068b2c19da8cb5e874bf9f6c}{gko::experimental::mpi::map\_rank\_to\_device\_id}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ comm,\ \doxymbox{\hyperlink{classgko_1_1CudaExecutor_aef0258494d14de0e56149b920c5173e5}{gko::CudaExecutor::get\_num\_devices}}());}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordflow}{return}\ \doxymbox{\hyperlink{classgko_1_1CudaExecutor_a075a1957241de65c0cad59f9534430a2}{gko::CudaExecutor::create}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ device\_id,\ gko::ReferenceExecutor::create());}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \}\},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \{\textcolor{stringliteral}{"{}hip"{}},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ [](MPI\_Comm\ comm)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordtype}{int}\ device\_id\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a0ec0c93a068b2c19da8cb5e874bf9f6c}{gko::experimental::mpi::map\_rank\_to\_device\_id}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ comm,\ \doxymbox{\hyperlink{classgko_1_1HipExecutor_a7b34cbd84692dafa2290e6630754aa97}{gko::HipExecutor::get\_num\_devices}}());}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordflow}{return}\ \doxymbox{\hyperlink{classgko_1_1HipExecutor_a8afc5210bf8dd8aedfceab4b73f0047b}{gko::HipExecutor::create}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ device\_id,\ gko::ReferenceExecutor::create());}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \}\},}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \{\textcolor{stringliteral}{"{}dpcpp"{}},\ [](MPI\_Comm\ comm)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordtype}{int}\ device\_id\ =\ 0;}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordflow}{if}\ (\doxymbox{\hyperlink{classgko_1_1DpcppExecutor_adf306338233c8e8aca0dec96aefe5d93}{gko::DpcppExecutor::get\_num\_devices}}(\textcolor{stringliteral}{"{}gpu"{}}))\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ device\_id\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a0ec0c93a068b2c19da8cb5e874bf9f6c}{gko::experimental::mpi::map\_rank\_to\_device\_id}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ comm,\ \doxymbox{\hyperlink{classgko_1_1DpcppExecutor_adf306338233c8e8aca0dec96aefe5d93}{gko::DpcppExecutor::get\_num\_devices}}(\textcolor{stringliteral}{"{}gpu"{}}));}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \}\ \textcolor{keywordflow}{else}\ \textcolor{keywordflow}{if}\ (\doxymbox{\hyperlink{classgko_1_1DpcppExecutor_adf306338233c8e8aca0dec96aefe5d93}{gko::DpcppExecutor::get\_num\_devices}}(\textcolor{stringliteral}{"{}cpu"{}}))\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ device\_id\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a0ec0c93a068b2c19da8cb5e874bf9f6c}{gko::experimental::mpi::map\_rank\_to\_device\_id}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ comm,\ \doxymbox{\hyperlink{classgko_1_1DpcppExecutor_adf306338233c8e8aca0dec96aefe5d93}{gko::DpcppExecutor::get\_num\_devices}}(\textcolor{stringliteral}{"{}cpu"{}}));}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \}\ \textcolor{keywordflow}{else}\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordflow}{throw}\ std::runtime\_error(\textcolor{stringliteral}{"{}No\ suitable\ DPC++\ devices"{}});}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \}}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textcolor{keywordflow}{return}\ \doxymbox{\hyperlink{classgko_1_1DpcppExecutor_af69a356ee07f641010df5d2e9e518bb0}{gko::DpcppExecutor::create}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ device\_id,\ gko::ReferenceExecutor::create());}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \}\}\};}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ exec\ =\ executor\_factory\_mpi.at(executor\_string)(MPI\_COMM\_WORLD);}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ num\_rows\ =\ grid\_dim;}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ partition\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(part\_type::build\_from\_global\_size\_uniform(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ exec-\/>get\_master(),\ comm.size(),}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \textcolor{keyword}{static\_cast<}GlobalIndexType\textcolor{keyword}{>}(num\_rows)));}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \doxymbox{\hyperlink{structgko_1_1matrix__data}{gko::matrix\_data<ValueType,\ GlobalIndexType>}}\ A\_data;}
\DoxyCodeLine{\ \ \ \ \doxymbox{\hyperlink{structgko_1_1matrix__data}{gko::matrix\_data<ValueType,\ GlobalIndexType>}}\ b\_data;}
\DoxyCodeLine{\ \ \ \ \doxymbox{\hyperlink{structgko_1_1matrix__data}{gko::matrix\_data<ValueType,\ GlobalIndexType>}}\ x\_data;}
\DoxyCodeLine{\ \ \ \ A\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a30716bd9e39a7beed52fc56d737fc07b}{size}}\ =\ \{num\_rows,\ num\_rows\};}
\DoxyCodeLine{\ \ \ \ b\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a30716bd9e39a7beed52fc56d737fc07b}{size}}\ =\ \{num\_rows,\ 1\};}
\DoxyCodeLine{\ \ \ \ x\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a30716bd9e39a7beed52fc56d737fc07b}{size}}\ =\ \{num\_rows,\ 1\};}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ range\_start\ =\ partition-\/>get\_range\_bounds()[rank];}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{const}\ \textcolor{keyword}{auto}\ range\_end\ =\ partition-\/>get\_range\_bounds()[rank\ +\ 1];}
\DoxyCodeLine{\ \ \ \ \textcolor{keywordflow}{for}\ (\textcolor{keywordtype}{int}\ i\ =\ range\_start;\ i\ <\ range\_end;\ i++)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \textcolor{keywordflow}{if}\ (i\ >\ 0)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ A\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a4f6be92270fa96bccdc502c83d248dc0}{nonzeros}}.emplace\_back(i,\ i\ -\/\ 1,\ -\/1);}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \}}
\DoxyCodeLine{\ \ \ \ \ \ \ \ A\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a4f6be92270fa96bccdc502c83d248dc0}{nonzeros}}.emplace\_back(i,\ i,\ 2);}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \textcolor{keywordflow}{if}\ (i\ <\ grid\_dim\ -\/\ 1)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ A\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a4f6be92270fa96bccdc502c83d248dc0}{nonzeros}}.emplace\_back(i,\ i\ +\ 1,\ -\/1);}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \}}
\DoxyCodeLine{\ \ \ \ \ \ \ \ b\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a4f6be92270fa96bccdc502c83d248dc0}{nonzeros}}.emplace\_back(i,\ 0,\ std::sin(i\ *\ 0.01));}
\DoxyCodeLine{\ \ \ \ \ \ \ \ x\_data.\doxymbox{\hyperlink{structgko_1_1matrix__data_a4f6be92270fa96bccdc502c83d248dc0}{nonzeros}}.emplace\_back(i,\ 0,\ \doxymbox{\hyperlink{namespacegko_a70dbe01ff95c7b953d3d737424c6feb5}{gko::zero<ValueType>}}());}
\DoxyCodeLine{\ \ \ \ \}}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ comm.synchronize();}
\DoxyCodeLine{\ \ \ \ ValueType\ t\_init\_end\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a508f97387781bb32dd2c7616dee1a786}{gko::experimental::mpi::get\_walltime}}();}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ A\_host\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(dist\_mtx::create(exec-\/>get\_master(),\ comm));}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ x\_host\ =\ dist\_vec::create(exec-\/>get\_master(),\ comm);}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ b\_host\ =\ dist\_vec::create(exec-\/>get\_master(),\ comm);}
\DoxyCodeLine{\ \ \ \ A\_host-\/>read\_distributed(A\_data,\ partition);}
\DoxyCodeLine{\ \ \ \ b\_host-\/>read\_distributed(b\_data,\ partition);}
\DoxyCodeLine{\ \ \ \ x\_host-\/>read\_distributed(x\_data,\ partition);}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ A\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(dist\_mtx::create(exec,\ comm));}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ x\ =\ dist\_vec::create(exec,\ comm);}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ b\ =\ dist\_vec::create(exec,\ comm);}
\DoxyCodeLine{\ \ \ \ A-\/>copy\_from(A\_host);}
\DoxyCodeLine{\ \ \ \ b-\/>copy\_from(b\_host);}
\DoxyCodeLine{\ \ \ \ x-\/>copy\_from(x\_host);}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ comm.synchronize();}
\DoxyCodeLine{\ \ \ \ ValueType\ t\_read\_setup\_end\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a508f97387781bb32dd2c7616dee1a786}{gko::experimental::mpi::get\_walltime}}();}
\DoxyCodeLine{}
\DoxyCodeLine{}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ local\_solver\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(bj::build().on(exec));}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ coarse\_solver\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ solver::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ .with\_preconditioner(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ schwarz::build().with\_local\_solver(local\_solver).on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ .with\_criteria(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::Iteration::build().with\_max\_iters(1000u).on(exec),}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::ResidualNorm<ValueType>::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_reduction\_factor(1e-\/7)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ .on(exec));}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ pgm\_fac\ =\ \doxymbox{\hyperlink{namespacegko_ae99fdd8bfa5ac694adcb7297d432967e}{gko::share}}(pgm::build().on(exec));}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{const}\ \doxymbox{\hyperlink{namespacegko_a107302a750bfa82a2107c77293f791c9}{gko::remove\_complex<ValueType>}}\ reduction\_factor\{1e-\/8\};}
\DoxyCodeLine{\ \ \ \ std::shared\_ptr<const\ gko::log::Convergence<ValueType>>\ logger\ =}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \doxymbox{\hyperlink{classgko_1_1log_1_1Convergence_ab0ec7b5f1f08c7237e2181e71cf7d5ed}{gko::log::Convergence<ValueType>::create}}();}
\DoxyCodeLine{\ \ \ \ std::shared\_ptr<gko::LinOp>\ Ainv\{\};}
\DoxyCodeLine{\ \ \ \ \textcolor{keywordflow}{if}\ (schw\_type\ ==\ \textcolor{stringliteral}{"{}two-\/level"{}})\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ Ainv\ =}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ solver::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_preconditioner(schwarz::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_local\_solver(local\_solver)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_coarse\_level(pgm\_fac)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_coarse\_solver(coarse\_solver)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_criteria(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::Iteration::build().with\_max\_iters(num\_iters).on(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ exec),}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::ResidualNorm<ValueType>::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_reduction\_factor(reduction\_factor)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ -\/>generate(A);}
\DoxyCodeLine{\ \ \ \ \}\ \textcolor{keywordflow}{else}\ \textcolor{keywordflow}{if}\ (schw\_type\ ==\ \textcolor{stringliteral}{"{}one-\/level"{}})\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ Ainv\ =}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ solver::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_preconditioner(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ schwarz::build().with\_local\_solver(local\_solver).on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_criteria(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::Iteration::build().with\_max\_iters(num\_iters).on(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ exec),}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::ResidualNorm<ValueType>::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_reduction\_factor(reduction\_factor)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ -\/>generate(A);}
\DoxyCodeLine{\ \ \ \ \}\ \textcolor{keywordflow}{else}\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ schw\_type\ =\ \textcolor{stringliteral}{"{}no-\/precond"{}};}
\DoxyCodeLine{\ \ \ \ \ \ \ \ Ainv\ =}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ solver::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_criteria(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::Iteration::build().with\_max\_iters(num\_iters).on(}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ exec),}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ gko::stop::ResidualNorm<ValueType>::build()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .with\_reduction\_factor(reduction\_factor)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec))}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ .on(exec)}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ -\/>generate(A);}
\DoxyCodeLine{\ \ \ \ \}}
\DoxyCodeLine{\ \ \ \ Ainv-\/>add\_logger(logger);}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ comm.synchronize();}
\DoxyCodeLine{\ \ \ \ ValueType\ t\_solver\_generate\_end\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a508f97387781bb32dd2c7616dee1a786}{gko::experimental::mpi::get\_walltime}}();}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ Ainv-\/>apply(b,\ x);}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ comm.synchronize();}
\DoxyCodeLine{\ \ \ \ ValueType\ t\_end\ =\ \doxymbox{\hyperlink{namespacegko_1_1experimental_1_1mpi_a508f97387781bb32dd2c7616dee1a786}{gko::experimental::mpi::get\_walltime}}();}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ res\_norm\ =\ \doxymbox{\hyperlink{namespacegko_ae3e355a655feea026a7c46814d37a9c9}{gko::as<vec>}}(logger-\/>get\_residual\_norm());}
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{auto}\ host\_res\ =\ \doxymbox{\hyperlink{namespacegko_adf2f8602dc9453752a787b40bbf13fcf}{gko::make\_temporary\_clone}}(exec-\/>get\_master(),\ res\_norm);}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \textcolor{keywordflow}{if}\ (comm.rank()\ ==\ 0)\ \{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ std::cout\ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nNum\ rows\ in\ matrix:\ "{}}\ <<\ num\_rows}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nNum\ ranks:\ "{}}\ <<\ comm.size()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nPrecond\ type:\ "{}}\ <<\ schw\_type}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nFinal\ Res\ norm:\ "{}}\ <<\ *host\_res-\/>get\_const\_values()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nIteration\ count:\ "{}}\ <<\ logger-\/>get\_num\_iterations()}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nInit\ time:\ "{}}\ <<\ t\_init\_end\ -\/\ t\_init}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nRead\ time:\ "{}}\ <<\ t\_read\_setup\_end\ -\/\ t\_init}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nSolver\ generate\ time:\ "{}}\ <<\ t\_solver\_generate\_end\ -\/\ t\_read\_setup\_end}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nSolver\ apply\ time:\ "{}}\ <<\ t\_end\ -\/\ t\_solver\_generate\_end}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ \textcolor{stringliteral}{"{}\(\backslash\)nTotal\ time:\ "{}}\ <<\ t\_end\ -\/\ t\_init}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <<\ std::endl;}
\DoxyCodeLine{\ \ \ \ \}}
\DoxyCodeLine{\}}

\end{DoxyCodeInclude}
 