Outage: Master LDAP (resolved)

Looks like I spoke too soon.. once was up, now is down. Sorry all.. I've alerted the engineers.

Looks like the last SVN server is back up -- I think we might officially be back up

Yes, we've been there; at an organization that doesn't have the availability they advertise, and those companies deservedly lose business. We aren't hacking on individuals, rather the organization that made claims that aren't met. We aren't saying people aren't doing the best with the resources they have currently, we are saying that after the organization already surpassed their downtime commitment in the first week of the year, they should have reset expectations or found other means to improve uptime.

SVN is still broken for me too

Svn works for me from the command line. However, if I want to browse the repository, it won't connect.

All, here is the update from our DIrector of Customer Service, Mohan:

Outage Duration- 15:28 UTC  – 15:35 UTC, 16:11 UTC – 18:08 UTC, 18:50 UTC - 19:09UTC  [ Total 2.13 hrs]
 
Problem statement –  Access to most of CloudForge services was affected due to LDAP server non availability in 3 different time periods between 15:28 – 19:09 UTC. The initial outage due to a Chassis failure in our datacenter had a ripple effect on our LDAP server accumulating too many connections which required the server to be rebooted. Our backup server due to misconfiguration, mirrored the problem faced by primary server. Recovery took more than expected time due to clean up needed in database.
 
Remediation - CollabNet corrected the configuration which will allow additional connection & avoid the problem in the future. Additionally CollabNet will also be work with the hosting service provider to re configure the server architecture with the aim to provide better fail over performance.
 
Regards,
Mohan Achar

Sorry all, obviously this has been a huge pain for everyone. We didn't expect the LDAP slave to fail when our main LDAP server was unreachable. Shannon is correct; there is no excuse. Unfortunately, a lot of the time was waiting for engineers to resolve this, and by the time I sent out an ETA update to the support staff the problem was solved roughly 15 minutes later. We apologize that it took so long to resolve something that could have been easily avoided.

Same here, SVN is still down.

Unfiortunately, this is not the first time.  I have been a customer since the "Dude" days and now I am looking elsewhere.

Was up for 5 mins for me, then down again.

same here..SVN is not still working for me . 

Git is also still down. 

Just saw some signs of life again....  time will tell if steady or not.

Edward,

Strange.. can you try again real quick? I'm hoping there just needed to be a services resync

SVN is still not responding for me

svn at commandline worked for me before, and is working now ... fingers crossed ...

All,

Our management team is in the process of creating a post-mortem, which I will post here once it is completed.

Still down for me, and my company.

Still no dice on my end. Tried comparing local vs repo... no response from SVN.

Its working for me now 12:30 PDT

Have a build I need to sort in the next hour to ship out heh. Would also be interested in how progress goes :)

Git is down for me, either!

Sorry for beating on a dead horse Jesse but is there any sign of a postmortem coming out?

3rd time -- signs of life observed.    First two only lasted ~5-10 mins each.

I'll withhold my confidence until 20-60 mins steady up-time.

Oh give 'em a break. They're working on it and when they're done, they'll let us know.

Haven't you ever been there? I know I have.

We've updated the status once again..I'm sorry to have misled. Apparently it was only up and healthy for a good 5-10 minutes. Our engineers are investigating as we type this.

Jesse Yowell

Support Engineer

0.1% is about 45min, so everybody should be requesting 7 free days because of this. I doubt they will drop below 97% though, it's about a day-long outage. But at that point I will not be looking for 2 free weeks of service, I will be switching.

I have performed several sync and commit operations. It's running smooth at the moment. 

Also, there's no excuse for not providing status updates when their core service is out. That's something that any person in the whole organization can do. Get the freaking owner on the web to come post here every half hour until it's fixed. Do it now.

Engineering said Main LDAP was brought back up. We're hoping it stays steady.. sorry for the eventful morning, everyone