Well you should worry about the type of food you eat. Whether you should worry about counting calories or not depends on what works for you.
Obviously you need to be in a calorie deficit to lose weight, but if by just cleaning up your diet, eating healthier, more protien and less-calorie dense...