Outage: Authentication Service (Fixed)

Date:  10 & 11 Feb 2014 UTC

Services Affected :  All the services are affected.

Description : We are experiencing a technical issue with our authentication server once again. Certain CloudForge users would experience difficulties to authenticate with the service. Our team members are working diligently to resolve the issue at the earliest. we will keep the posted updated on the progress. 

 

Update: Our authentication servers sustained the restoration solution we attempted only for few minutes. The good news is that we discovered the cause of the problem to be corrupted LDAP data. we have immediately kick started a new LDAP server and are currently performing a clean data import from our database. During the import operation customers will be experiencing connection errors  due to restarts performed in regular intervals. [Posted at Feb 11 2:00 AM UTC]

 

ETA for completion of import: 4 hrs [6 AM UTC]. 

 

We will provide a detailed Root Cause Analysis statement and remediation plan upon restoration of service. Apologies for the service disruption & the inconvenience caused. 

 

Update: We have completed the import successfully and the services are back and running. Feel free to raise tickets for any further issues.

Have more questions? Submit a request

59 Comments

  • 0
    Avatar
    Dwamian Mcleish

    Cmon guys!

  • 0
    Avatar
    Erik Dillingham

    Seriously?

  • 0
    Avatar
    Nathan Crouther

    Please provide followup on the root cause and what is being done to prevent a recurrence.  It seems that the authentication service has been a frequent source of outages, so I would like to know that there is a plan in place to understand the underlying cause of the problems and prevent similar problems in the future.

  • 0
    Avatar
    Brandon Wolfe

    This is absolutely ridiculous.  This has helped us make the decision to jump ship as quickly as possible.... https://www.atlassian.com/software/bitbucket/overview

  • 0
    Avatar
    Vincent Dee

    OMG not again.....

  • 0
    Avatar
    Nazmi Doganc

    seriously, it is midnight my time and I gotta commit the code, what should I do? Unbelievable...

  • 0
    Avatar
    Dwamian Mcleish

    Any updates?

     

  • 0
    Avatar
    Vincent Dee

    bump.

  • 0
    Avatar
    Erik Dillingham

    Can we get some kind of ETA, please?

  • 0
    Avatar
    Subramanian Vairavan

    Our Authentication servers are still experiencing issues and we are working on this issue with high Priority. Our estimated ETA is 30 minutes to resolution.

     

    Thanks,

    Support Team.

  • 0
    Avatar
    Nathan Crouther

    Here is an example of a great root cause analysis that inspires confidence as a customer:  http://aws.amazon.com/message/65648/

    Hopefully we can get a similar report on the authentication server problems.

  • 0
    Avatar
    Steve An

    still not working.

  • 0
    Avatar
    Ross Mallace

    Still not working.

  • 0
    Avatar
    Dmytro BAZULIN

    appears to be working again ATM

  • 0
    Avatar
    Michael Foltz

    Not working for me.

  • 0
    Avatar
    Sean Barnes

    Been getting "Unable to connect to a repository at URL" "Unexpected HTTP status 500 'Internal Server Error' on '/base'".. etc.  This related?

  • 0
    Avatar
    Subramanian Vairavan

    GIT and Bugzilla service are available now. We are still working on SVN issue, we will keep you updated on the status.

     

    Thanks,

    Support Team.

  • 0
    Avatar
    Sean Barnes

    Ok.

  • 0
    Avatar
    Vincent Dee

    Working for me either.

  • 0
    Avatar
    Alan Hoffman

    I need to get access to my code.... how long is this going to take?

  • 0
    Avatar
    Dwamian Mcleish

    Guys/Gals...let's cheer them on instead... obviously being frustrated just isn't helping. Cloudforge team, we are depending on you... I know you can make it happen...let's fix these frequent outages. . I will stay patient today...but these outages have to stop... you are professionals! ...... *slow clap*

  • 0
    Avatar
    Subramanian Vairavan

    Hi All,

     

    We are sorry for the inconvenience caused because of this issue. SVN Service is also restored now. Please try accessing the repository and service and feel free to file a support ticket if any issues.

     

    Thanks,

    Support Team.

  • 0
    Avatar
    Ross Mallace

    Did you say the issue is resolved? I am still getting a server 500 error when trying to access my repository. 

    Thanks

  • 0
    Avatar
    Matt Luthi

    Not working for me yet: 

    SharpSvn.SvnRepositoryIOException: Unable to connect to a repository at URL 'https://hdsdevelopment.svn.cvsdude.com/finalcut' ---> SharpSvn.SvnRepositoryIOException: Unexpected HTTP status 500 'Internal Server Error' on '/finalcut'
    ---> SharpSvn.SvnRepositoryIOException: Additional errors: ---> SharpSvn.SvnRepositoryIOException: OPTIONS request on '/finalcut' failed: 500 Internal Server Error
    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at SharpSvn.SvnClientArgs.HandleResult(SvnClientContext client, SvnException error, Object targets)
    at SharpSvn.SvnClientArgs.HandleResult(SvnClientContext client, svn_error_t* error, Object targets)
    at SharpSvn.SvnClient.Update(ICollection`1 paths, SvnUpdateArgs args, SvnUpdateResult& result)
    at SharpSvn.SvnClient.Update(String path, SvnUpdateArgs args, SvnUpdateResult& result)
    at Ankh.Commands.SolutionUpdateCommand.PerformUpdate(CommandEventArgs e, ProgressWorkerArgs wa, SvnRevision rev, Boolean allowUnversionedObstructions, Boolean updateExternals, Boolean setDepthInfinity, IEnumerable`1 groups, SvnUpdateResult& updateResult)
    at Ankh.Commands.SolutionUpdateCommand.<>c__DisplayClass9.<OnExecute>b__6(Object sender, ProgressWorkerArgs a)
    at Ankh.ProgressRunnerService.ProgressRunner.Run(Object arg)

     

  • 0
    Avatar
    Vincent Dee

    Was working for like 3 minutes then all of a sudden "SSL handshake failed: Connection reset by peer" ....

  • 0
    Avatar
    Alan Hoffman

    It was working for a few minutes... now getting the same error as before!

  • 0
    Avatar
    Vincent Dee

    Hmm anyone else having an issue where their old credentials doesn't work anymore? Just tried committing and it all of a sudden asked me for my credentials, but they don't work anymore..........

  • 0
    Avatar
    Ross Mallace

    Same issue here. Prompting for credentials and they are not working.

  • 0
    Avatar
    Mansour Behabadi

    The issue is still present. Can't pull, commit or even look at the logs.

  • 0
    Avatar
    William Nguyen

    It seems to have forgotten my SVN password (TortoiseSVN).

    Can't seem to get back in no matter what password I use.

Article is closed for comments.