Lots of experienced paddlers have organized races with multiple classes or been participants in some. It's not rocket science. Most of it is judge's discretion on race day when you see how many of what type of boat you wind up with. Generally, most that I've been to have a 3 boat rule to make a separate class. You can also make separate classes based on age and gender. Usually it's dependent on how many awards you have to give away.
If you want a good turn out, the key is promotion and advertising. Create a race website with online registration. Next, try to pick a date which doesn't have a lot of other conflicting events which could siphon off racers. Last, pray for good weather.