Each sector has one wall gap at the center of each side. You may pass through this to the next sector. If you go through the gap at the edge of the gameboard, you will re-enter the board on the opposite side.
If you destroy a wall (or pass through it with magic) at the edge of the board, you will likewise re-enter the equivalent square on the opposite side of the board. The half-wall shown on the other sector does not stop your movement.
The exception to all this is Portals. There are three pairs of Portals: two each of red, green, and blue. They are set up outside the board, as part of the initial layout, and do not move if the board is rearranged. If you go out a green Portal, you come in the other green Portal. If you go out a red Portal, you come in the other red Portal.
Note that the two- and four-player boards use only two pairs of portals, and the five-player board uses none at all.