If I understand this correctly, that is common. The cutting teeth in the middle of the blade do not contact the file as much as the outer ones do. Concentrate on keeping the blade flat on the file, easier when blade is off the drill. Keep in mind that the portion of the tooth that cuts is the very tip. The very tip of all the teeth should show signs of being filed while sharpening, if not, then keep filing til they do. This actually may be by design to allow "relief" for materials being cut to escape. Don't be afraid to file the upper edge as well, just keep the same angle that is currently there, providing that is the correct angle. Sometimes I mark the teeth with a sharpie marker to make sure I'm at the proper angle. From my experience, if you keep the proper angle when filing you won't do any harm to your blades, this includes the pilot point as well, which I like to file parallel to the cutting edge in short strokes and not filing perpendicular to it. Hope this helps.