We employ a fourth-order compact finite difference scheme (FOS) with the multigrid algorithm to solve the three dimensional Poisson equation. We test the influence of different orderings of the grid space and different grid-transfer operators on the convergence and efficiency of our high accuracy algorithm. Fourier smoothing analysis is conducted to show that FOS has a smaller smoothing factor than the traditional second-order central difference scheme (CDS). A new method of Fourier smoothing analysis is proposed for the partially decoupled red-black Gauss-Seidel relaxation with FOS. Numerical results are given to compare the computed accuracy and the computational efficiency of FOS with multigrid against CDS with multigrid.