Outage: Authentication Service (Fixed)

Date:  10 & 11 Feb 2014 UTC

Services Affected :  All the services are affected.

Description : We are experiencing a technical issue with our authentication server once again. Certain CloudForge users would experience difficulties to authenticate with the service. Our team members are working diligently to resolve the issue at the earliest. we will keep the posted updated on the progress. 

 

Update: Our authentication servers sustained the restoration solution we attempted only for few minutes. The good news is that we discovered the cause of the problem to be corrupted LDAP data. we have immediately kick started a new LDAP server and are currently performing a clean data import from our database. During the import operation customers will be experiencing connection errors  due to restarts performed in regular intervals. [Posted at Feb 11 2:00 AM UTC]

 

ETA for completion of import: 4 hrs [6 AM UTC]. 

 

We will provide a detailed Root Cause Analysis statement and remediation plan upon restoration of service. Apologies for the service disruption & the inconvenience caused. 

 

Update: We have completed the import successfully and the services are back and running. Feel free to raise tickets for any further issues.

59 Comments

  • 0
    Avatar
    jose martinez

    i'm still having problems accessing any svn related services.  :(

  • 0
    Avatar
    Dwamian Mcleish

    Guys/Gals...let's cheer them on instead... obviously being frustrated just isn't helping. Cloudforge team, we are depending on you... I know you can make it happen...let's fix these frequent outages. . I will stay patient today...but these outages have to stop... you are professionals! ...... *slow clap*

  • 0
    Avatar
    Sean Barnes

    Been getting "Unable to connect to a repository at URL" "Unexpected HTTP status 500 'Internal Server Error' on '/base'".. etc.  This related?

  • 0
    Avatar
    Erik Dillingham

    Seriously?

  • 0
    Avatar
    Gopal Raichur

    Still no access to SVN. This is the second time this is happening in span of a week. Any updated ETA on when this will be resolved?

  • 0
    Avatar
    Portal Admin

    @ jose thanks - corrected

  • 0
    Avatar
    Vincent Dee

    Was working for like 3 minutes then all of a sudden "SSL handshake failed: Connection reset by peer" ....

  • 0
    Avatar
    Subramanian Vairavan

    GIT and Bugzilla service are available now. We are still working on SVN issue, we will keep you updated on the status.

     

    Thanks,

    Support Team.

  • 0
    Avatar
    Erik Dillingham

    Can we get some kind of ETA, please?

  • 0
    Avatar
    Chirag Surti

    still no access to svn

  • 0
    Avatar
    Portal Admin

    @Jason - Thanks for your patronage. We are committed to giving you the golden days/time back real soon.

  • 0
    Avatar
    Vincent Dee

    Hmm anyone else having an issue where their old credentials doesn't work anymore? Just tried committing and it all of a sudden asked me for my credentials, but they don't work anymore..........

  • 0
    Avatar
    Alan Hoffman

    I need to get access to my code.... how long is this going to take?

  • 0
    Avatar
    Subramanian Vairavan

    Our Authentication servers are still experiencing issues and we are working on this issue with high Priority. Our estimated ETA is 30 minutes to resolution.

     

    Thanks,

    Support Team.

  • 0
    Avatar
    Ashwin K V Vemula

    still same problem for me. Please let me know any steps to perform

  • 0
    Avatar
    Dwamian Mcleish

    Any updates? we are still waiting to push a critical fix. 

  • 0
    Avatar
    Dwamian Mcleish

    ok..4 hours? no more paitence... migrating off your platform as soon as I can get access back to our repo. 

  • 0
    Avatar
    Ross Mallace

    Did you say the issue is resolved? I am still getting a server 500 error when trying to access my repository. 

    Thanks

  • 0
    Avatar
    Subramanian Vairavan

    Hi All,

     

    We are sorry for the inconvenience caused because of this issue. SVN Service is also restored now. Please try accessing the repository and service and feel free to file a support ticket if any issues.

     

    Thanks,

    Support Team.

  • 0
    Avatar
    Cijo Sunny

    Is there any progress???? 

  • 0
    Avatar
    Ross Mallace

    Understood Mansour. This outage has been going on for too long. Guys why is there no fail-over  LDAP server?

  • 0
    Avatar
    Matt Luthi

    Not working for me yet: 

    SharpSvn.SvnRepositoryIOException: Unable to connect to a repository at URL 'https://hdsdevelopment.svn.cvsdude.com/finalcut' ---> SharpSvn.SvnRepositoryIOException: Unexpected HTTP status 500 'Internal Server Error' on '/finalcut'
    ---> SharpSvn.SvnRepositoryIOException: Additional errors: ---> SharpSvn.SvnRepositoryIOException: OPTIONS request on '/finalcut' failed: 500 Internal Server Error
    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at SharpSvn.SvnClientArgs.HandleResult(SvnClientContext client, SvnException error, Object targets)
    at SharpSvn.SvnClientArgs.HandleResult(SvnClientContext client, svn_error_t* error, Object targets)
    at SharpSvn.SvnClient.Update(ICollection`1 paths, SvnUpdateArgs args, SvnUpdateResult& result)
    at SharpSvn.SvnClient.Update(String path, SvnUpdateArgs args, SvnUpdateResult& result)
    at Ankh.Commands.SolutionUpdateCommand.PerformUpdate(CommandEventArgs e, ProgressWorkerArgs wa, SvnRevision rev, Boolean allowUnversionedObstructions, Boolean updateExternals, Boolean setDepthInfinity, IEnumerable`1 groups, SvnUpdateResult& updateResult)
    at Ankh.Commands.SolutionUpdateCommand.<>c__DisplayClass9.<OnExecute>b__6(Object sender, ProgressWorkerArgs a)
    at Ankh.ProgressRunnerService.ProgressRunner.Run(Object arg)

     

  • 0
    Avatar
    Alan Hoffman

    It was working for a few minutes... now getting the same error as before!

  • 0
    Avatar
    Chakravarthy Shankar

    Hi,

    We have completed the import successfully and the services are back and running. Feel free to raise tickets for any further issues. We will provide a detailed Root Cause Analysis statement and remediation plan upon restoration of service. Apologies for the service disruption & the inconvenience caused. 

     

    Thanks,

    Chakravarthy.

  • 0
    Avatar
    Matt Luthi

    @Mansour - I can work on my code right now. I just can't check it into the main repository :) But it is a hassle - we can't do a build now which prevents/delays our testing run.

  • 0
    Avatar
    Mansour Behabadi

    The issue is still present. Can't pull, commit or even look at the logs.

  • 0
    Avatar
    Ross Mallace

    Same issue here. Prompting for credentials and they are not working.

  • 0
    Avatar
    Dwamian Mcleish

    Cmon guys!

  • 0
    Avatar
    Portal Admin

    @Ross: We do have a fail-over LDAP, however the data is corrupted in both the servers since they are both synced real time hence the need for us to do a fresh import.

  • 0
    Avatar
    Dwamian Mcleish

    Try restarting your IDE or svn client. Usually clears it up for me

Article is closed for comments.