Nidacore was mentioned earlier. So was Airex. There are others as well, but this section should not be read as an endorsement of one over the other. My experience is with the two that were mentioned so I'm going to speak from that perspective.
Honeycomb cores are the next generation of core materials and, even though they are both costly and not always readily available, I think these are the first evolution of fiberglass construction that is, hands down, a distinct improvement over standard organic cores.
The honeycomb material is made from polypropylene which has good points and bad points.
Polypropylene is plastic. It's the same plastic used to make extruded parts and cheap children's toys. It's also spun, braided, and used to make two ropes and safety lines for boats because it's stretchy and it floats. Understanding these qualities of polypropylene should help you understand it's qualities as a core material.
First let's look at the mechanical aspects of these cores. They are honeycombed, but why is that good or bad? Well, a hexagonal support structure between two fiberglass skins makes for a rigid sandwich that is protected from shearing in three directions.
The small hexagonal voids also become individual pockets in the structure. This means that if the boat should become punctured or water should penetrate the outer skin through improperly bedded hardware, the moisture will be confined to the compromised cell or cells and won't wick through the entire core like it would with organic materials.
Because these cores are mostly air surrounded by a latticework of I-beams, they are also very light. Sandwiching a honeycombed core material to the backside of an unsupported solid fiberglass panel, such as the fore deck of a small runabout, will greatly improve the structure without adding much weight.
Honeycombed cores have excellent impact resistance too. Because it's made out of a semi-elastic plastic, a sharp impact to a cored material will first cause the core to compress, then spring back to its original shape. These are the reasons this core material is so prevalent in many hull constructions.
How about for transoms? This is where they're not recommended. A transom, as reiterated countless times so far, requires high shear strength and high compressive strength. Polypropylene cores have multi-directional shear strength but it's not particularly high. It also has fairly low compressive strength.
Another factor at play with these materials is the layering necessary for excessively thick panels. A transom is often the thickest panel on a boat because it requires the greatest strength and rigidity.
Honeycombed cores over an inch thick can become weak and collapse. When using these cores on thick panels, it's necessary to layup two successive layers of core material separated by a central layer of fiberglass, essentially turning the sandwich construction into something closer to a club sandwich construction; outer skin – core – central skin – core – inner skin.
This does provide and extra layer of protection if the panel should be punctured, but it is labor intensive and, in the end, over-engineered and no better than using an alternative core material.