Not done it, but just edit that rule to restrict the IP address, i.e.
allow 1.2.3.4; # Allow your IP
deny all; # Deny everything else
Not 100% sure if purge via the plugin will appear as your IP or the server IP - you can either allow both or temporarily comment out the log line to check.
Create a custom conf when editing WO config files though to prevent your changes getting lost on upgrading WO.