Outage: Authentication Service (Fixed)

Date:  10 & 11 Feb 2014 UTC

Services Affected :  All the services are affected.

Description : We are experiencing a technical issue with our authentication server once again. Certain CloudForge users would experience difficulties to authenticate with the service. Our team members are working diligently to resolve the issue at the earliest. we will keep the posted updated on the progress. 

 

Update: Our authentication servers sustained the restoration solution we attempted only for few minutes. The good news is that we discovered the cause of the problem to be corrupted LDAP data. we have immediately kick started a new LDAP server and are currently performing a clean data import from our database. During the import operation customers will be experiencing connection errors  due to restarts performed in regular intervals. [Posted at Feb 11 2:00 AM UTC]

 

ETA for completion of import: 4 hrs [6 AM UTC]. 

 

We will provide a detailed Root Cause Analysis statement and remediation plan upon restoration of service. Apologies for the service disruption & the inconvenience caused. 

 

Update: We have completed the import successfully and the services are back and running. Feel free to raise tickets for any further issues.

Have more questions? Submit a request

59 Comments

  • 0
    Avatar
    Anand Rajakumaran

    Yeah there is some problem in authentication.Keeps prompting for password.

     

  • 0
    Avatar
    Maya Gadde

    Same here, keeps prompting for a password.

  • 0
    Avatar
    Dwamian Mcleish

    Try restarting your IDE or svn client. Usually clears it up for me

  • 0
    Avatar
    Subramanian Vairavan

    Our authentication server sustained restoration only for few minutes. The good news is that we discovered the cause of the problem to be corrupted LDAP entries. We have immediately kick started a new LDAP server & performing a clean data import from our database to the new LDAP server. The ETA for completion of service will be 4 hrs.

     

    Thanks,

    Support Team

  • 0
    Avatar
    Dwamian Mcleish

    ok..4 hours? no more paitence... migrating off your platform as soon as I can get access back to our repo. 

  • 0
    Avatar
    Mansour Behabadi

    Yet another example why SVN should be abandoned (ie s@%$ does go south no matter what you do but with git, at least you can commit and move forward). Why am I here? Not my choice, my employer's.

  • 0
    Avatar
    Ross Mallace

    As Subramanian stated: Its an LDAP issue not a SVN issue. GIT can and is often configured to use LDA. Get your facts right before blaming SVN.

  • 0
    Avatar
    Mansour Behabadi

    I'm not blaming SVN because of this instance, I'm blaming it for being centralized. It doesn't matter what you configure git on the remote server, I can continue working on my local repo no matter what catastrophic event happens on the remote repo, be it LDAP, OS or god forbid, fire and earthquake issues.

  • 0
    Avatar
    Ross Mallace

    Understood Mansour. This outage has been going on for too long. Guys why is there no fail-over  LDAP server?

  • 0
    Avatar
    Matt Luthi

    @Mansour - I can work on my code right now. I just can't check it into the main repository :) But it is a hassle - we can't do a build now which prevents/delays our testing run.

  • 0
    Avatar
    Mansour Behabadi

    Yes, there's always a workaround. Mine's good ol' email and patches. But like you said, hassle.

  • 0
    Avatar
    Portal Admin

    @Ross: We do have a fail-over LDAP, however the data is corrupted in both the servers since they are both synced real time hence the need for us to do a fresh import.

  • 0
    Avatar
    chen ma

    Cannot login, always fail on SVN login till now...

  • 0
    Avatar
    Saurabh Chowdhury

    I just woke up into this mess, just checking what time zone is the correct point of reference for me. It definitely ain't UTC as says on the post title :-)

  • 0
    Avatar
    jose martinez

    i'm still having problems accessing any svn related services.  :(

  • 0
    Avatar
    Portal Admin

    @Saurabh - I updated the description to reflect the correct timzone/time

     

  • 0
    Avatar
    jose martinez

    Please note its 2AM UTC on Feb 11th, not the 10th.

  • 0
    Avatar
    Portal Admin

    @ jose thanks - corrected

  • 0
    Avatar
    Jason Cipriani

    Thank you for clearly describing the problem so far and for being straightforward and professional. I have been with you guys for 6 years and I am always impressed with the handling of the rare outage. Recent months have been rocky but I'm sure you will return to you standard level of reliable service soon. It is unfortunate that this is occurring but I wish other companies would handle it in a similar way. I hope you get it fixed soon so we can all start using the service again.

  • 0
    Avatar
    Jason Cipriani

    (Actually, 8 years... not sure where the time went.)

  • 0
    Avatar
    Portal Admin

    @Jason - Thanks for your patronage. We are committed to giving you the golden days/time back real soon.

  • 0
    Avatar
    Gopal Raichur

    Still no access to SVN. This is the second time this is happening in span of a week. Any updated ETA on when this will be resolved?

  • 0
    Avatar
    Portal Admin

    @Gopal - Our ETA is to complete the import by 6 AM UTC[90 mins from current time]. 

  • 0
    Avatar
    Jesus Gutierrez

    Still no access to SVN. 

  • 0
    Avatar
    Dwamian Mcleish

    Any updates? we are still waiting to push a critical fix. 

  • 0
    Avatar
    Chirag Surti

    still no access to svn

  • 0
    Avatar
    Cijo Sunny

    Is there any progress???? 

  • 0
    Avatar
    Chakravarthy Shankar

    Hi,

    We have completed the import successfully and the services are back and running. Feel free to raise tickets for any further issues. We will provide a detailed Root Cause Analysis statement and remediation plan upon restoration of service. Apologies for the service disruption & the inconvenience caused. 

     

    Thanks,

    Chakravarthy.

  • 0
    Avatar
    Ashwin K V Vemula

    still same problem for me. Please let me know any steps to perform

Article is closed for comments.