Hi Feather, you'll have to bear with me, as it's easier to show someone than write it.
Ok look at the foot, at theback a nipple hook a filler thread on, at front right in the centre on the leading edge a gripper for anchoring the filler cord, like crochet thread or old fashioned button hole twist thread. Directly behind the filler thread gripper, is the needle hole. If you have the clp on type of buttonhole attachment, slide the foot back using your finger, release it and it shoots back to it's start position at the front. Most problems with automatic buttonholers is the foot has not been in it's start position on the fabric.
Because the foot is so ungainly it's very easy when you're positioning those red cross hairs on the foot to the marker on the fabric, to miss noticing that slider foot has slid backwards a fraction. Just that fraction alone will cause an incorrect stitch out.
Another problem area is asking the foot to climb over a lump, by that I mean seam allowance, shaping dart, another buttonhole if you are doing vertical buttonholes.
How thick or thin your layers of fabric, interlining, interfacings are, if you are doing a structured jacket with all it's layers of shaping interlinings, a bound buttonhole is a better option. If it's a blouse it could be the addition of a button hole stabiliser mounted between the fabric layers, and a square of tearaway stabilser under the fabric will create a good crisp buttonhole.
Are you doing the sensor or auto buttonhole, as the sensor buttonhole needs the sensor arm pulled right down and positioned correctly. The Auto buttonhole doesnt use the sensor arm. Your machine manual will tell you if you have the non sensor option, normally you set the parameters for the first one and then the machine will repeat it for the rest of the buttonholes, or you tell it, new size.
Youmay need to change the density of your zigzag, also the width, play with the settings on sandwiches of scrap cloth first, till you find a setting that works. I'd like to say that the default setting for the buttonholer is fine, but Janome can't make a one off setting to accomodate every type of fabric, the varying thicknesses we use and whether you use filler threads or not. So it really is a case of practise first, then do them on the garment.