So many technical variables (woods, pot size, strings, heads, bridges, tailpieces), and so many different sets of ears and aural expectations. I've tried many variations of the above, and I always come back to my ideal setup for my everyday banjos: Fiberskyn head, medium gauge phosphor bronze strings, Bacon tone ring, medium moon bridge, 11" maple pot.

Getting nylon tone out of steel strings: in my experience putting a little piece of foam on the play side of the bridge will give you a wonderfully mellow tone...and it sounds very nylon-y. Putting some variation of tape over the bridge can help too. some use gaffers, some use masking...lots of types of tape. I like foam best....in fact I use foam all over my banjos...in nut slots, for tailpiece cushion, in the rim...I have a whole drawer of foam that I have found all over the world. ---- Some banjos I have found can go either way, but many want one or the other. Some banjos just don't sound good with nylon no matter what you do....some banjos should never have steel on them, for structural and sound reasons. ==== I find that I only like working with nylon when I have basic violin style friction tuners. I have an old Buckbee that was never "updated" in any way. Those tuners work great! but nylon on modern tuners is always a hastle for me. ----they can sound great tho on many banjos and there is something there....but if you want to flirt with many sonic variations I recommend steel and a little foam, here & there...

I have Minstrel Nylgut strings on a Stone banjo (no metal tone ring, just the wooden pot) and it sounds wonderful! It is lowered down from open G to open E and i use it mostly to back my voice at the local Farmer's market. And yep, nylon/plastic strings can sound like wet noodles on some banjos. Head tension and bridge can be critical set up factors for securing the tone you desire. banjered

Modern banjos often need the nut slots filing to be the correct fit for synthetic strings. Not a difficult job but one you are unlikely to carry out on a new banjo unless you know for certain that synthetic strings are what you’re going to be playing.

If you tried nylon/nylgut strings with a typical modern ebony-capped bridge which probably came with your banjo, that also won’t have helped an appraisal either. Nylon or gut generally produce their best on an uncapped, thin and much lighter weight bridge. The tailpiece also has a significant effect. Every element of set-up has some effect.

I responded because Nylon or gut strings can be bright, even with a distinctive 'crackle'. It's really only a cliche that they are innately mellow or ‘plunky’. I use gut or rectified nylon for fingerstyle. I avoid Nylgut. For clawhammer I prefer steel, including for fretless clawhammer playing.

Experimenting with tone and feel is part of the fun. That said if you have tone that you like, that’s great. There are no rules. But there is set-up know-how to get the optimal tone from synthetic strings.

You say you tried a variety of Nylgut and nylon, but in your post you only mention Nylgut. Nylon is a different beast and ranges from the ultra-light (see LaBella's "Classic 17" pack) to the kind of high-tension strings used by classical guitarists. Not to mention fluorocarbon strings, which are relatively higher tension at the same gauge (or smaller gauge at the same tension, depending on how you want to look at it).

I've never been crazy about Nylguts. The plunk-y, hollow sound they seem to produce is nice on occasion but I wouldn't want my own banjo to sound like that all of the time.

Ditto what EEB said about bridges.

I have an older Ome Tupelo strung with Aquila Red 11b and a John Balch hide head.
It is my favorite playing around the house banjo.
Not sure of the tone your looking for, but this setup works for me for clawhammer/thumb lead.

I tried 11-26-11 steel strings and liked the deep plunk. I'll probably try Gold Tone 13-30 steel strings on an A-scale.

Modern banjos are essentially plectrum banjos.

The original gut strung banjo had a neck set with no back angle. The fingerboard was level and even with the head. With a 1/2" tall bridge the action was 3/16" to 1/4" and gave plenty of clearance for the strings to vibrate without buzzing.

When pick playing became popular, wire stings were found to be better for that. The major change to banjos for plectrum playing was to add back angle to the neck. This lowered the action (wire strings do not move as much) and raised the bridge to provide more clearance to keep the pick from scratching the head.

Putting polyester strings (what "nylgut" are made of) on a modern steel string banjo usually requires a taller bridge, 3/4" to 1" high, otherwise the strings will buzz if played with any confidence.

Then there is the subject of bridges. Attached are images of gut era bridges along with modern bridges in common use today. The difference is striking!