Marcel --- You have posed a very interesting question!  Normally, a tenor banjo is strung and tuned in the key of C, with CGDA tuning, and it's not a transposing instrument like a trumpet or clarinet is.  (When a C major chord is played on a banjo, that's what is heard.  But as you probably know, when a C note is played on a trumpet or clarinet, a Bb note is what is heard.)

So when playing with horns in a band, it's normally the horns that have to transpose by changing the key one full tone/step up (because the horns are Bb instruments).  They usually do this "on the fly," changing notes mentally from the lead sheet or sheet music as they play.  For example, if a song is in the key of F, the horns need to play it in the key of G so that their melody notes as heard will match those from the piano or vocalist.  If the band is using charts, the horn parts will often be transposed into the higher key so the musicians can just play what's written.  That may or may not be the case with a lead sheet.  

But your Bb F C G tuning turns this arrangement on its head and makes the banjo a Bb transposing instrument (by lowering the tuning a full step).  That's fine if the horns are playing from the sheet music, but it's going to mess up the pianist, the vocalist and any other instrument that's not in the key of Bb -- when they play/sing the written notes, they'll be a full step higher than you and the horns.   

I guess you could work around this problem by giving them a transposed version of the sheet music.  So for example, if the concert key of the song is in G for the horns and you, the vocalist and piano would need a version in F -- which is the key that the song will actually be played/sounded in.  With the digital versions that are now available for most tunes, it's usually easy to get a song instantly transposed into whatever key you need. 

But I do think you're probably "paddling against the current" with this unusual tuning arrangement (not to mention the problem of needing heavier gauge strings and a slightly higher bridge to compensate for the slack tuning).  Since the fingering patterns would be exactly the same with standard CGDA tuning, my suggestion is that you stick with that standard tuning and simply retrain your brain as to which chords fall onto which frets.  Then you can leave the transposing issue to the horns, who either are (or should be) used to dealing with it. 

 I hope these two possible solutions are helpful to you, maybe other posters will have some more ideas.  SETH   

Years ago I played with that tuning with regular tenor strings on a plectrum banjo…

I have tried your tuning, and liked it for the mellower tone. Playing melody in keys like C, and G are not as comfortable.
I would think the string gauges for Bb F C G tuning would be 32 ,24 ,16, 10 or 11. Jim