--- title: "Solver benchmarks" resource_files: - figures/gap_vs_target.png - figures/time_vs_target.png output: rmarkdown::html_vignette fontsize: 11pt bibliography: references.bib vignette: > %\VignetteIndexEntry{Solver benchmarks} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- The `prioriactions R` package offers diverse optimization solvers, both academic and non-academic, to tackle mathematical models. Notably, it supports cutting-edge commercial optimization software such as `gurobi` and `CPLEX`, as well as non-academic alternatives like `CBC` and `symphony`. In the realm of supported solvers, `CPLEX` and `gurobi` generally stand out for their superior speed. It's worth noting that while these are commercial tools, they do offer special academic licenses at no charge. Depending on the intricacy of the management planning problem at hand, solvers based on open-source software may only marginally trail behind their commercial counterparts in terms of the speed and quality of the solutions. In the following analysis, we will explore and evaluate the performance of these different software options that `prioriactions` provides. ## Experimental settings This analysis encompassed the generation of prioritizations through all solvers available in `prioriactions`, accompanied by measuring the time and quality of the solutions (*gap*) each solver required to complete the process. We explored a set of management planning problems characterized by distinct models (referred to as models **(A)**, **(B)**, **(C)**). Among the models considered, **(C)** stands out for being the largest model due to the incorporation of spatial connectivity requirements between actions ($blm_k$ = 0.8). In contrast, model **(B)** entails connectivity solely between management units ($blm$ = 0.8), while model **(A)** lacks any form of spatial requirement. Consequently, solving model **(A)** should be comparatively straightforward, given its absence of spatial constraints (further elaborated in Table below with model sizes). Also, we introduce variations in the target achievement levels (10\%, 20\%, 30\%, and 40\% respectively), thereby altering the inherent attributes of the models. All benchmark scenarios encompassed the Mitchell River case study presented, i.e. 2,316 planning units, 45 features, and 4 threats. ```{r table1, tidy=FALSE, echo=FALSE} df <- data.frame(Modelling_Setting = c("(A) $blm = 0, blm_k = 0$", "(B) $blm = 0.8, blm_k = 0$", "(C) $blm = 0, blm_k = 0.8$"), Variables = c(43180, 61428, 89150), Constraints = c(37371, 92115, 175281), Nonzeros = c(136566, 264302, 458356)) knitr::kable( df, booktabs = TRUE, caption = 'Model size (Variables, Constraints, Nonzeros elements in the constraint matrix) for different modelling scenarios.' ) ``` Our experimental setup was conducted on a machine powered by an Intel Core i5-1240P 3.30GHz processor, paired with 16 GB of RAM LPDDR5-6400MHz, and running Ubuntu 22.04 LTS. Firstly, the `time_limit` parameter, which designates the maximum period the solver dedicates to pursuing an optimal solution, was set to 10,800 seconds, equivalent to a span of 3 hours. It is noteworthy that no constraint was placed on the *gap* to achieve (`gap_limit` = 0}), underscoring the aim of attaining the utmost optimal solution within the aforementioned three-hour timeframe. Secondly, the number of cores utilized was set to ten. Futhermore, we employed the most recent versions of all available optimizers within the `prioriactions` framework. Specifically, for `BM CPLEX`, we utilized version 22.1.1.0 in conjunction with `Rcplex` version 0.3.7. `Gurobi` was employed at version 10.0.0, while `Rcbc` was utilized at its version 0.1.0.9001. Additionally, for `symphony`, we leveraged `Rsymphony` with version 0.1-33 to ensure a comprehensive and up-to-date foundation for our analyses. To derive these outcomes, we solely adjusted in the third step, involving the alteration of the solver's name within the `solve()` function embedded in the `prioriactions` package. ```r # solve the model solution <- solve(model, solver = "gurobi", gap_limit = 0, time_limit = 10800, solution_limit = FALSE, cores = 10, output_file = TRUE, verbose = TRUE) ``` In the subsequent segments, we will offer distinct log outputs corresponding to each of the experimental solvers operating within scenario **(A)**, where the recovery target is set at 10%. These log records will serve as illustrative examples, providing insight into the information presented throughout the optimization process. ### `Gurobi` The logs follow a consistent structure, albeit with variations based on the information being presented and the manner of presentation. This structure can be succinctly summarized into four key sections: **(i) optimizer settings**; the initial part includes details about the settings and configurations of the optimizer (lines 1 to 8). **(ii) presolve**; to fine-tune the model. This step involves searching for the bounds associated with the relaxed solution (infeasible), among other tasks (lines 15 to 31). **(iii) solutions search**; entails an active pursuit of the optimal solution (or one of the optimal solutions, if multiple exist, lines 33 to 61). And, **(iv) summary**; summarises the entire optimization process, providing quality *gaps* and times, among other things (lines 63 to 69). In this particular instance, the optimal solution (with a *gap* of 0.00%) was successfully attained, yielding a value of 646, 10 seconds after the process started. ``` 1 Gurobi 10.0.0 (linux64, R) logging started dom 06 ago 2023 19:49:56 2 3 Set parameter Username 4 Set parameter TimeLimit to value 10800 5 Set parameter MIPGap to value 0 6 Set parameter NodefileStart to value 0.5 7 Set parameter LogFile to value "modelA_10_gurobi_log.txt" 8 Set parameter Threads to value 10 9 Academic license - for non-commercial use only - expires 2024-01-25 10 Gurobi Optimizer version 10.0.0 build v10.0.0rc2 (linux64) 11 12 CPU model: 12th Gen Intel(R) Core(TM) i5-1240P, instruction set [SSE2|AVX|AVX2] 13 Thread count: 16 physical cores, 16 logical processors, using up to 10 threads 14 15 Optimize a model with 37371 rows, 43180 columns and 136566 nonzeros 16 Model fingerprint: 0x96123de4 17 Variable types: 34965 continuous, 8215 integer (8215 binary) 18 Coefficient statistics: 19 Matrix range [3e-01, 4e+00] 20 Objective range [1e+00, 1e+00] 21 Bounds range [1e+00, 1e+00] 22 RHS range [5e-01, 2e+02] 23 Found heuristic solution: objective 7947.0000000 24 Found heuristic solution: objective 4978.0000000 25 Presolve removed 35017 rows and 35259 columns 26 Presolve time: 0.15s 27 Presolved: 2354 rows, 7921 columns, 66284 nonzeros 28 Variable types: 0 continuous, 7921 integer (7904 binary) 29 Found heuristic solution: objective 817.0000000 30 31 Root relaxation: objective 5.657750e+02, 3822 iterations, 0.05 seconds (0.26 work units) 32 33 Nodes | Current Node | Objective Bounds | Work 34 Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time 35 36 0 0 565.77500 0 279 817.00000 565.77500 30.7% - 0s 37 H 0 0 699.0000000 565.77500 19.1% - 0s 38 H 0 0 663.0000000 565.77500 14.7% - 0s 39 H 0 0 651.0000000 565.77500 13.1% - 0s 40 0 0 573.58333 0 399 651.00000 573.58333 11.9% - 0s 41 0 0 575.33333 0 387 651.00000 575.33333 11.6% - 0s 42 0 0 577.54167 0 457 651.00000 577.54167 11.3% - 1s 43 0 0 580.01543 0 429 651.00000 580.01543 10.9% - 1s 44 H 0 0 647.0000000 580.51543 10.3% - 1s 45 0 0 580.51543 0 434 647.00000 580.51543 10.3% - 1s 46 0 0 580.51543 0 431 647.00000 580.51543 10.3% - 1s 47 0 0 590.16667 0 350 647.00000 590.16667 8.78% - 1s 48 0 0 590.26667 0 334 647.00000 590.26667 8.77% - 1s 49 0 0 590.26667 0 327 647.00000 590.26667 8.77% - 1s 50 0 0 599.85185 0 258 647.00000 599.85185 7.29% - 1s 51 0 0 603.87037 0 236 647.00000 603.87037 6.67% - 1s 52 0 0 614.47222 0 161 647.00000 614.47222 5.03% - 1s 53 0 0 614.47222 0 158 647.00000 614.47222 5.03% - 1s 54 0 0 614.72222 0 130 647.00000 614.72222 4.99% - 1s 55 0 0 614.72222 0 146 647.00000 614.72222 4.99% - 1s 56 0 0 614.72222 0 194 647.00000 614.72222 4.99% - 1s 57 0 0 614.72222 0 160 647.00000 614.72222 4.99% - 2s 58 H 0 0 646.0000000 614.72222 4.84% - 2s 59 0 2 614.72222 0 160 646.00000 614.72222 4.84% - 2s 60 1966 2055 630.50000 123 31 646.00000 626.33333 3.04% 9.7 5s 61 7481 7363 630.50000 460 43 646.00000 626.33333 3.04% 7.2 10s 62 63 Explored 8192 nodes (87667 simplex iterations) in 11.58 seconds (16.40 work units) 64 Thread count was 10 (of 16 available processors) 65 66 Solution count 8: 646 647 651 ... 7947 67 68 Optimal solution found (tolerance 0.00e+00) 69 Best objective 6.460000000000e+02, best bound 6.460000000000e+02, gap 0.0000% ``` ### `CPLEX` Similar to `gurobi`, the `CPLEX` log adheres closely to the outlined structure. Precisely, lines 3 to 9 correspond to **(i)**, while lines 10 to 30 encompass **(ii)**. Furthermore, lines 31 to 49 represent **(iii)**, and lines 51 to 63 represent **(iv)**. In this context, the optimal solution was swiftly identified within a mere 2.08 seconds while maintaining the identical objective value of 646. ``` 1 CPLEX environment opened 2 Warning: The following options are not availables using cplex solver: output_file 3 Rcplex: num variables=43180 num constraints=37371 4 Version identifier: 22.1.1.0 | 2022-11-28 | 9160aff4d 5 CPXPARAM_TimeLimit 10800 6 CPXPARAM_MIP_Tolerances_AbsMIPGap 0 7 CPXPARAM_MIP_Tolerances_MIPGap 0 8 CPXPARAM_MIP_Pool_RelGap 0 9 CPXPARAM_MIP_Pool_AbsGap 0 10 Found incumbent of value 8215.000000 after 0.00 sec. (2.35 ticks) 11 Tried aggregator 4 times. 12 MIP Presolve eliminated 1402 rows and 300 columns. 13 MIP Presolve added 2688 rows and 0 columns. 14 MIP Presolve modified 354 coefficients. 15 Aggregator did 34959 substitutions. 16 Reduced MIP has 3698 rows, 7921 columns, and 67628 nonzeros. 17 Reduced MIP has 7904 binaries, 17 generals, 0 SOSs, and 0 indicators. 18 Presolve time = 0.39 sec. (1525.39 ticks) 19 Probing time = 0.00 sec. (2.02 ticks) 20 Tried aggregator 1 time. 21 Detecting symmetries... 22 Reduced MIP has 3698 rows, 7921 columns, and 67628 nonzeros. 23 Reduced MIP has 7904 binaries, 17 generals, 0 SOSs, and 0 indicators. 24 Presolve time = 0.03 sec. (51.39 ticks) 25 Probing time = 0.01 sec. (2.02 ticks) 26 Clique table members: 5591. 27 MIP emphasis: balance optimality and feasibility. 28 MIP search method: dynamic search. 29 Parallel mode: deterministic, using up to 16 threads. 30 Root relaxation solution time = 0.06 sec. (98.92 ticks) 31 Nodes Cuts/ 32 Node Left Objective IInf Best Integer Best Bound ItCnt Gap 33 34 * 0+ 0 7947.0000 0.0000 100.00% 35 * 0+ 0 925.0000 0.0000 100.00% 36 * 0+ 0 768.0000 0.0000 100.00% 37 0 0 566.4167 353 768.0000 566.4167 2719 26.25% 38 * 0+ 0 766.0000 566.4167 26.06% 39 0 0 577.0000 387 766.0000 Cuts: 453 3327 24.67% 40 * 0+ 0 651.0000 577.0000 11.37% 41 0 0 581.0833 361 651.0000 Cuts: 384 3778 10.74% 42 0 0 592.9111 293 651.0000 Cuts: 356 4484 8.92% 43 * 0+ 0 646.0000 592.9111 8.22% 44 0 0 -1.00000e+75 0 646.0000 592.9111 4484 8.22% 45 0 0 604.6852 207 646.0000 Cuts: 279 5004 6.40% 46 0 0 608.2222 170 646.0000 Cuts: 189 5264 5.85% 47 0 0 637.3333 13 646.0000 Cuts: 161 5393 1.34% 48 0 0 cutoff 646.0000 5406 0.00% 49 Elapsed time = 2.07 sec. (2723.02 ticks, tree = 0.01 MB, solutions = 1) 50 51 Clique cuts applied: 447 52 Cover cuts applied: 1269 53 Mixed integer rounding cuts applied: 3 54 Zero-half cuts applied: 56 55 56 Root node processing (before b&c): 57 Real time = 2.08 sec. (2724.47 ticks) 58 Parallel b&c, 16 threads: 59 Real time = 0.00 sec. (0.00 ticks) 60 Sync time (average) = 0.00 sec. 61 Wait time (average) = 0.00 sec. 62 ------------ 63 Total (root+branch&cut) = 2.08 sec. (2724.47 ticks) ``` ### `CBC` In the case of `CBC` solver, the log provides an even higher specificity level than its counterparts in `gurobi` and `CPLEX`. In this instance, the breakdown is as follows: lines 5 to 11 correspond to **(i)**, offering initial insights. Lines 12 to 66 delve into more intricate detailed aspects of presolve **(ii)**. Subsequently, lines 67 to 80 represent **(iii)**. Further along, lines 81 to 110 illustrate **(iv)**. Note that we have employed a stopping criterion set at 3600 seconds. This decision stems from the understanding that with this non-academic solver, an optimal solution might not be attained within a shorter timeframe (as it indeed was the case). This approach ensures that we are able to present the complete structure of the log output comprehensively. Thus, an optimal solution was not reached (*gap* of 0.04%). Instead, we found a solution with an objective value of 653, marking a distinction from the previous solvers’ outcomes, which converged at 646. ``` 1 Welcome to the CBC MILP Solver 2 Version: 2.10.5 3 Build Date: Apr 21 2021 4 5 command line - problem -threads 10 -log 1 -verbose 15 -ratio 0 -sec 10800 -timem elapsed 6 -heuristicsOnOff on -solve -quit (default strategy 1) 7 verbose was changed from 0 to 15 8 ratioGap was changed from 0 to 0 9 seconds was changed from 1e+100 to 10800 10 Option for timeMode changed from cpu to elapsed 11 Option for heuristicsOnOff changed from off to on 12 Continuous objective value is 563.542 - 8.95 seconds 13 Cgl0004I processed model has 37313 rows, 42882 columns (7930 integer (7914 of which binary)) 14 and 136210 elements 15 Cbc0038I Initial state - 372 integers unsatisfied sum - 123.042 16 Cbc0038I Pass 1: (9.34 seconds) suminf. 79.66667 (242) obj. 604.667 iterations 720 17 Cbc0038I Pass 2: (9.36 seconds) suminf. 78.16667 (239) obj. 606.167 iterations 3 18 Cbc0038I Pass 3: (9.39 seconds) suminf. 71.16667 (218) obj. 620.167 iterations 21 19 Cbc0038I Pass 4: (9.41 seconds) suminf. 63.83333 (196) obj. 634.833 iterations 22 20 Cbc0038I Pass 5: (9.44 seconds) suminf. 56.16667 (173) obj. 650.167 iterations 23 21 Cbc0038I Pass 6: (9.46 seconds) suminf. 46.50000 (144) obj. 669.5 iterations 29 22 Cbc0038I Pass 7: (9.49 seconds) suminf. 38.16667 (119) obj. 686.167 iterations 25 23 Cbc0038I Pass 8: (9.52 seconds) suminf. 33.83333 (106) obj. 694.833 iterations 13 24 Cbc0038I Pass 9: (9.54 seconds) suminf. 25.50000 (81) obj. 711.5 iterations 25 25 Cbc0038I Pass 10: (9.57 seconds) suminf. 18.16667 (59) obj. 726.167 iterations 22 26 Cbc0038I Pass 11: (9.59 seconds) suminf. 13.16667 (44) obj. 736.167 iterations 15 27 Cbc0038I Pass 12: (9.62 seconds) suminf. 7.50000 (27) obj. 747.5 iterations 17 28 Cbc0038I Pass 13: (9.64 seconds) suminf. 1.16667 (7) obj. 761.167 iterations 20 29 Cbc0038I Solution found of 767 30 Cbc0038I Relaxing continuous gives 767 31 Cbc0038I Cleaned solution of 767 32 Cbc0038I Before mini branch and bound, 7554 integers at bound fixed and 31499 continuous 33 Cbc0038I Full problem 37313 rows 42882 columns, reduced to 262 rows 253 columns 34 Cbc0038I Mini branch and bound improved solution from 767 to 765 (9.83 seconds) 35 Cbc0038I Round again with cutoff of 744.104 36 Cbc0038I Pass 14: (9.91 seconds) suminf. 79.66667 (242) obj. 604.667 iterations 0 37 Cbc0038I Pass 15: (9.94 seconds) suminf. 73.50000 (225) obj. 615.5 iterations 17 38 Cbc0038I Pass 16: (9.96 seconds) suminf. 64.83333 (199) obj. 632.833 iterations 26 39 Cbc0038I Pass 17: (9.99 seconds) suminf. 60.50000 (186) obj. 641.5 iterations 13 40 Cbc0038I Pass 18: (10.01 seconds) suminf. 56.16667 (173) obj. 650.167 iterations 13 41 Cbc0038I Pass 19: (10.04 seconds) suminf. 47.83333 (148) obj. 666.833 iterations 25 42 Cbc0038I Pass 20: (10.07 seconds) suminf. 39.50000 (123) obj. 683.5 iterations 25 43 Cbc0038I Pass 21: (10.09 seconds) suminf. 31.16667 (98) obj. 700.167 iterations 25 44 Cbc0038I Pass 22: (10.12 seconds) suminf. 25.83333 (82) obj. 710.833 iterations 16 45 Cbc0038I Pass 23: (10.14 seconds) suminf. 19.16667 (62) obj. 724.167 iterations 20 46 Cbc0038I Pass 24: (10.17 seconds) suminf. 9.83333 (34) obj. 742.833 iterations 28 47 Cbc0038I Pass 25: (10.21 seconds) suminf. 10.69574 (36) obj. 744.104 iterations 206 48 Cbc0038I Pass 26: (10.24 seconds) suminf. 10.12908 (36) obj. 744.104 iterations 48 49 Cbc0038I Pass 27: (10.28 seconds) suminf. 9.67908 (35) obj. 744.104 iterations 144 50 Cbc0038I Pass 28: (10.31 seconds) suminf. 10.69574 (34) obj. 744.104 iterations 26 51 Cbc0038I Pass 29: (10.34 seconds) suminf. 10.67908 (37) obj. 744.104 iterations 16 52 Cbc0038I Pass 30: (10.36 seconds) suminf. 10.67908 (37) obj. 744.104 iterations 3 53 Cbc0038I Pass 31: (10.62 seconds) suminf. 38.57092 (112) obj. 744.104 iterations 1444 54 Cbc0038I Pass 32: (10.67 seconds) suminf. 37.19574 (109) obj. 744.104 iterations 231 55 Cbc0038I Pass 33: (10.70 seconds) suminf. 36.59574 (107) obj. 744.104 iterations 159 56 Cbc0038I Pass 34: (10.71 seconds) suminf. 36.59574 (107) obj. 744.104 iterations 0 57 Cbc0038I Pass 35: (10.74 seconds) suminf. 36.52908 (107) obj. 744.104 iterations 29 58 Cbc0038I Pass 36: (10.77 seconds) suminf. 36.26241 (106) obj. 744.104 iterations 82 59 Cbc0038I Pass 37: (10.80 seconds) suminf. 36.19574 (106) obj. 744.104 iterations 115 60 Cbc0038I Pass 38: (10.83 seconds) suminf. 35.92908 (105) obj. 744.104 iterations 91 61 Cbc0038I Pass 39: (10.86 seconds) suminf. 35.52908 (105) obj. 744.104 iterations 58 62 Cbc0038I Pass 40: (10.89 seconds) suminf. 35.26241 (103) obj. 744.104 iterations 48 63 Cbc0038I Pass 41: (10.91 seconds) suminf. 35.26241 (103) obj. 744.104 iterations 0 64 Cbc0038I Pass 42: (10.94 seconds) suminf. 34.59574 (102) obj. 744.104 iterations 65 65 Cbc0038I Pass 43: (11.00 seconds) suminf. 33.39574 (99) obj. 744.104 iterations 253 66 Cbc0038I Rounding solution of 764 is better than previous of 765 67 68 Cbc0038I Before mini branch and bound, 7332 integers at bound fixed and 30532 continuous 69 Cbc0038I Full problem 37313 rows 42882 columns, reduced to 1758 rows 1877 columns 70 Cbc0038I Mini branch and bound did not improve solution (15.54 seconds) 71 Cbc0038I After 15.54 seconds - Feasibility pump exiting with objective of 742 - took 6.36 seconds 72 Cbc0012I Integer solution of 742 found by feasibility pump after 0 iterations and 0 nodes 73 (15.55 seconds) 74 Cbc0027I Exiting on user event 75 Cbc0005I Partial search - best objective 653 (best possible 629.9), took 3188627 iterations 76 and 1068 nodes (3615.33 seconds) 77 Cbc0032I Strong branching done 16196 times (715305 iterations), fathomed 0 nodes and fixed 0 78 variables 79 Cbc0035I Maximum depth 52, 0 variables fixed on reduced cost 80 Cuts at root node changed objective from 565.042 to 629.9 81 Probing was tried 1274 times and created 211829 cuts of which 4667 were active after adding 82 rounds of cuts (4.965 seconds) 83 Gomory was tried 1274 times and created 9045 cuts of which 0 were active after adding rounds 84 of cuts (32.725 seconds) 85 Knapsack was tried 1274 times and created 233 cuts of which 0 were active after adding rounds 86 of cuts (22.010 seconds) 87 Clique was tried 10 times and created 0 cuts of which 0 were active after adding rounds of cuts 88 (0.012 seconds) 89 MixedIntegerRounding2 was tried 1274 times and created 20769 cuts of which 0 were active after 90 adding rounds of cuts (12.944 seconds) 91 FlowCover was tried 10 times and created 150 cuts of which 0 were active after adding rounds 92 of cuts (0.263 seconds) 93 TwoMirCuts was tried 1274 times and created 50422 cuts of which 0 were active after adding 94 rounds of cuts (11.078 seconds) 95 ZeroHalf was tried 1274 times and created 921 cuts of which 0 were active after adding rounds 96 of cuts (10.462 seconds) 97 ImplicationCuts was tried 20 times and created 634 cuts of which 0 were active after adding 98 rounds of cuts (0.006 seconds) 99 100 Result - User ctrl-cuser ctrl-c 101 102 Objective value: 653.00000000 103 Lower bound: 629.900 104 Gap: 0.04 105 Enumerated nodes: 1068 106 Total iterations: 3188627 107 Time (CPU seconds): 3620.45 108 Time (Wallclock seconds): 3620.51 109 110 Total time (CPU seconds): 3620.46 (Wallclock seconds): 3620.52 ``` ### `Symphony` Contrary to the previous logs shown, `symphony` offers a minimalistic yet essential amount of information. As in `CBC`, we have set the stop criteria to 3600 seconds, for the same reasons. As noted, sections **(i)** and **(ii)** are omitted, while section **(iv)** only indicates the objective value of the solution achieved (lines 38 to 44). In turn, the section **(iii)** is similar to the log from `gurobi` and `CPLEX` (lines 8 to 36). Note that the most recent solution uncovered has a target value of 646. While it is currently evaluating whether this constitutes the optimal solution (reflecting a *gap* of 2.49%), it is pertinent to highlight that we are confident in asserting its optimality based on the solutions identified by `gurobi` and `CPLEX`. ``` 1 Starting Preprocessing... 2 Preprocessing finished... 3 coefficients modified: 268 4 constraints removed: 48 5 variables fixed: 290 6 variables aggregated: 17 7 8 Solving... 9 10 Time Done Queued LB UB Gap 11 48 1 1 629.90 652.00 3.39 12 54 8 8 629.90 652.00 3.39 13 60 17 17 629.90 652.00 3.39 14 66 26 26 629.90 652.00 3.39 15 72 36 36 629.90 651.00 3.24 16 78 47 47 629.90 649.00 2.94 17 84 61 61 629.90 648.00 2.79 18 90 76 76 629.90 648.00 2.79 19 96 90 90 629.90 648.00 2.79 20 102 106 106 629.90 648.00 2.79 21 ... 22 3511 4343 4343 629.90 646.00 2.49 23 3517 4348 4348 629.90 646.00 2.49 24 3523 4352 4352 629.90 646.00 2.49 25 3529 4357 4357 629.90 646.00 2.49 26 3535 4362 4362 629.90 646.00 2.49 27 3541 4366 4366 629.90 646.00 2.49 28 3547 4369 4369 629.90 646.00 2.49 29 3553 4373 4373 629.90 646.00 2.49 30 3559 4378 4378 629.90 646.00 2.49 31 3566 4382 4382 629.90 646.00 2.49 32 3572 4386 4386 629.90 646.00 2.49 33 3578 4389 4389 629.90 646.00 2.49 34 3585 4394 4394 629.90 646.00 2.49 35 3592 4398 4398 629.90 646.00 2.49 36 3598 4402 4402 629.90 646.00 2.49 37 38 **************************************************** 39 * Time Limit Reached * 40 **************************************************** 41 42 43 Solution Found: Node 217, Level 109 44 Solution Cost: 646.000000000 ``` ## Results We have conducted 48 tests, encompassing three distinct models (models **(A)**, **(B)**, and **(C)**) and spanning four recovery targets (10%, 20%, 30%, and 40%). These tests were carried out across all four available solvers within the `prioriactions` framework, comprising two academic solvers (`gurobi` and `CPLEX`) and two non-academic alternatives (`CBC` and `symphony`). ![](../vignettes/figures/gap_vs_target.png){width="100%"} ![](../vignettes/figures/time_vs_target.png){width="100%"} ### Academic solvers: `gurobi` vs `CPLEX` Upon comparing the two academic solvers, a similarity in behaviour becomes evident. In the context of `gurobi`, optimal solutions were successfully achieved across all scenarios, as evidenced by **gaps** amounting to zero. Similarly, when employing `CPLEX`, we observed zero *gaps* across all scenarios, with the exceptions occurring in model **(C)** targeting 10%, where a gap of 0.73% emerged, and in model **(C)** targeting 30%, which exhibited a *gap* of 0.24% (see upper right box in previous Figure). Nonetheless, a consistent disparity becomes apparent in terms of processing times. `gurobi` consistently exhibited a lead in verifying optimal solutions across most scenarios, with only slight variations in model **(A)** (distanced by seconds). Assuming that `CPLEX` achieved optimal solutions in all instances, verification required an average of 3510.847 seconds. In contrast, `gurobi` outperformed with an average of 1365.47 seconds, representing approximately 39% of the time taken by `CPLEX` (see bottom boxes in the previous Figure). ### Non-academic solvers: `CBC` vs `symphony` In this context, it is a more varied panorama, with `CBC` notably standing out. In 8 out of the 12 comparative analyses against `symphony`, `CBC` demonstrated superior solution qualities, as indicated by the upper boxes in Figure above. Furthermore, the average *gaps* achieved by `CBC` stand at 6.35%, while its counterpart averages 9.06%. Turning to run times, `CBC` exhibited the capability to verify optimal solutions ahead of `symphony` in model **(A)**, particularly with target restrictions of 30% and 40%. However, it is noteworthy that both solvers encountered challenges in validating the optimality of the obtained solutions in the remaining models (see bottom boxes in Figure above). ## Conclusions The benchmark findings underscore the considerable variability in the time required to resolve multi-action management planning challenges. This temporal difference is contingent upon diverse factors, encompassing the problem's scale, intricacy, and the solver engaged in formulating the prioritization. Notably, specific solvers, such as the `symphony` solver, may demand several hours to address a predicament that could be resolved within minutes by other solvers like `CPLEX` or `gurobi.` In terms of recommendations, we advise prioritizing the utilization of `gurobi` and `CPLEX` solvers whenever feasible, as they frequently exhibit superior performance with respect to the non-academic solvers. While academics can avail special licenses for these solvers at no cost, it is crucial to recognize that conservation planners within governmental or non-governmental entities might not have access to these resources. In scenarios where access to the aforementioned solvers is limited, we propose the adoption of the `CBC` solver, which generally demonstrates better efficacy compared to `symphony.` In any case, due to the complexity of exploring in detail all the possible instances (number of units, species, threats and their respective combinations), we recommend being cautious with these results. Considering the intrinsic combinatorial nature of these models, even solvers as robust as `gurobi` may encounter difficulties when tackling exhaustive analyses.