Fedora OpenID issues resolved

It is very likely that you have seen the issues we had with logging in to Fedora Infrastructure services, or other websites that use Fedora OpenID to authenticate you.

The issue presented itself during login, where as soon as you hit the Login button to submit your username and password, it would spin and wait for sometimes minutes, and retrying the entire process from start
would "magically" work most of the time.

I am very excited to tell you that we have found the root cause of this issue, and it has now been resolved.

If you still have problems with Fedora OpenID, either this issue or any other, please file a Fedora Infra ticket at Fedora Infra trac, or ping me on IRC (puiterwijk on FreeNode).

What happened

First, a very little bit of background: in Fedora Infrastructure, we have multiple reverse proxies (at the time of writing 10), which are used to actually request our websites. So when you request any of the Fedora Infra services,
your request ends up at any of those 10 reverse proxies, which then passes the request on through our internal network.

Now, Ipsilon authorizes your username and password with the Fedora Account System.
But for accessing that, it also uses our reverse proxies.

Now, we have two of these proxies in our main datacenter where all of our servers are hosted.
However, since we had no special case for the servers there, it could be that Ipsilon would try to access FAS to verify your user through a proxy all the way over in Europe, which is at the very least suboptimal.

Another part of the issue was that we have two proxies that are on donated machines wich are a little underpowered, and as such could not carry the load that was thrown at them.

So the fix that we have applied is to add a special case for our servers in the main datacenter to always use the two reverse proxies that are local to that datacenter.
This means that to verify your username, the request will stay within the datacenter and not cross the big pool of water at least twice.

Easy fix, but it had been slightly tricky to find the cause.

N.B.: The reason this was not a problem before Ipsilon was because we had exactly this special case in when we were using FedOAuth, but I had forgot to transfer it during the migration to Ipsilon.

Fixing Gnash setting system volume to 100%

Domain held hostage, DNSSEC to the rescue