Home  CFD-ACE+ Tips Table Methods for Domain Decomposition in CFD-ACE+
|
|
Methods for Domain Decomposition in CFD-ACE+ |
|
|
Introduction:
The CFD-ACE+ solver can be run in parallel mode by using
domain
decomposition, whereby
you divide up the data among multiple processors. The number of
zones
is equated to the number of processes or in other words, each processor
would solve for a single zone of the overall mesh.
The parallel feature of CFD-ACE+ provides several methods and options
for
decomposition of the domain. In this tip we will talk about these
methods and options and their usage. In many cases, the manner of
decomposition
influences the parallel efficiency and speed-up. For better speed-ups
it is important that the workload is balanced between processors, in
other words, all the processors gets nearly equal number of
cells. The user
has to
use his own intuition as to which decomposition would be most
appropriate for his/her simulation before attempting to do the domain
decomposition.
Syntax:
The domain decomposition in CFD-ACE+ is done by executing the
dtf_decompose script. When you type dtf_decompose at the command
prompt, you can get its full
usage.
The syntax for the command is:
dtf_decompose [-version] [-metis | -cell_groups | -orig_topo | -x | -y
| -z | -material | -wavefront] [-even] [-combine] [-keepFF] [-file_out
outFile.DTF]
[-restart] inFile.DTF sim# num_procs
Method 1 #
-metis
This is the default method for dtf_decompose. So if the user does
not
provide any other method, -metis option is used. METIS is
multi-level graph partitioning scheme used for partitioning
unstructured meshes. It provides high quality partitions with
less CPU
time. This decomposition method does workload balancing.
Depending on the number of partitions, METIS uses two
different
ways of partitions.
-- K-way Partitioning for N > 8
-- Recursive Bisection for N < 8
For more details on these partitioning schemes, the user can refer to
the following web site. http://www-users.cs.umn.edu/~karypis/metis/
The syntax is:
$
dtf_decompose -metis model.DTF sim# num_procs
where, sim# is the simulation number and, num_procs refer to the number
of processors, which is equal to number of decomposed zones.
For example,
$
dtf_decompose -metis test_metis.DTF 1 4
This indicates that simulation # 1 would be decomposed into 4
zones. The
decomposed domain is stored in simulation # 2, which is stored in the
same DTF file. When you open the file in CFD-ACE-GUI or CFD-VIEW,
it prompts the user select the appropriate simulation number in the DTF
file.
Figure 1 shows the original interface between two zones, and Figure 2
shows the new interfaces created after decomposition into four zones
Method 2 # -cell_groups
This method can be used to decompose the domain along cell
groups. So
if a simulation has 4 cell groups, it will decompose the simulation
into 4 zones. Decomposing by cell groups may be useful for
special
cases such as materials grouping. Note that since this is
done by cell groups, there is no attempt to load balance the zones.
Method
3 # -orig_topo
This method allows the user to partition the domains along the original
topology as it was created in CFD-GEOM. If the user has taken
care of
load balancing in GEOM, this method can be used. Note that since
the decomposition is done along the original zones, there is no attempt
to load balance the zones.
Figure 3 shows the same original topology after decomposition.
Method 4 # -x, -y, -z decomposition
This method allows the user to decompose the domain along the X, Y or
Z-axis.
In this case, the decomposition is workload balanced. In other
words, the
decomposition
will be done in such a way that the number of cells for each processor
is nearly equal.
Figure
4 shows the original domain with 2 zones, one is coarse and the
other one is refined. After decomposing this into 4 zones, in
Figure 5, we can see that
the z decomposition is workload balanced as there are more number of
cuts in the refined zone.
Method 5 # -material
In this method, the decomposition will take place along the material
interfaces. So if your simulation domain contains fluid and solid
zones, then this method would allow you to group all the fluid zones
into one decomposed zone and all the solid zones into another
decomposed zones. Note that since the
decomposition is done along the material zones, there is no attempt to
load balance the zones.
Figure 6 shows the original domain and Figure 7 shows the domain after
decomposition into 2 zones (i.e. solid and fluid zone).
Method 6 # -wavefront
This method is based on the Wavefront Element Reordering scheme, which
can be used to improve the efficiency of the global matrix
decomposition process performed during an analysis. It minimizes
the
number of operations needed to decompose the assembled global matrices
during an analysis. This method is used rarely. This
decomposition
method provides workload balancing.
There are some other options, which are described below, that can
be used with the above methods.
Option 1 # -even
This option allows the user to do an even x, y or z decomposition,
which means it will be spatially balanced (not workload
balanced). In other words, the
decomposition
will be done in such a way that the number of cells for each processor
is nearly equal. Note that this option is only used with x, y or
z
decomposition.
Figure 8, 9, and 10 shows example of even –x, -y, -z
decompositions.
Option 2 # -combined
This option will allow the user to combine the Volume and Boundary
Conditions
with identical properties to be decomposed into one
zone. This option reduces the number of volume and boundary
conditions after
decomposition and can be used with any method described
above.
Option 3 # -keepFF
This option allows the user to keep the original Fluid-Fluid interfaces
inside of
a zone. This becomes useful if the user would like to retain the
Fluid-Fluid interface to output Mass Flow Summary or the Heat Flow
Summary at the interface. This option can be
used with any method described above.
Option 4 # -restart
This option can be used with other methods, when the user would like to
do a restart from the serial solution. It provides mapping of
face data from a simulation to the decomposed simulation, which is
needed for restarts.
In addition to the above options in V2004, we can run FSI (Fluid
Structure Interaction) cases in
parallel. This is done by decomposing all the stress volume
conditions
into one zone and decomposing all the other zones (fluid + non-stress
solid zones) into (n-1) processors. The decomposition for FSI
cases
is
done in the same way as illustrated above. The decomposition algorithm
identifies the stress volume conditions automatically and decomposes it
into one domain.
This summarizes all the methods and options that can be used for
running
dtf_decompose prior to submitting the case in parallel by running the
cfd-ace-mpi script.
If you have any questions about this feature or would like
us to discuss some other topic in the future, please let us know.
Amit Saxena
Customer Support Engineer
ESI Group
|
|
|
