Lower Draw Distance gives worse FPS

Summary:
The title sums it up pretty well, lowering the Draw Distance option gives lower FPS and sometimes does nothing at all

Steps to reproduce:

  1. Start a world
  2. Go to options and lower the Draw Distance option
  3. ???
  4. Profit Lower FPS

Expected Results:
One would expect the mentioned FPS to be higher when you have lower Draw Distance

Actual Results:
The results are quite the opposite of what one would expect

Notes:
I get around 70-80 sometimes 60 when playing with full Draw Distance but rarely goes beyond 65 when Draw Distance is at its lowest

Attachments:
Not really needed is it?

Version Number and Mods in use:
Version 1.0, same results with or without mods

System Information:
MSI GTX 970 3.5GB
Intel Core i7-4790K 4GHz
16GB RAM
Windows 7