More than a day after bitcoin wallet provider Blockchain.info first indicated it was experiencing a severe outage, its services have been fully restored, a company blog post has revealed.
Issues began with Blockchain noting that it had suspended SharedCoin transactions in what was initially suspected to be a short delay. However, that announcement, made on 17th March, proceeded a long period of downtime that extended into the late evening the next day.
Blockchain updated users that all Blockchain services were operational again as of 10pm PST, 18th March.
As of 10pm PT, 5am UTC Blockchain services are RESTORED http://t.co/IssrKmpKRI Thank you for your patience and support
— Blockchain.info (@blockchain) March 19, 2014
The company had originally indicated that the problems stemmed from an ‘unusual database issue’, though the problem was later revealed to be the result of an error that affected several tables meant to track information contained in the bitcoin blockchain ledger.
Blockchain stated that though the problem was easy to identify, fixing the issue proved challenging due to the size of the affected data sets.
Explained Blockchain’s security and distributed systems expert Andreas Antonopoulos:
“We decided to pursue multiple methods to try to fix the problem, including re-compiling a custom version of MySQL to get around the error, while simultaneously restoring databases from our backup systems onto new servers. Because of the large dataset, each step in testing, restoring or copying data can take a very long time, making the entire process slow.”
Blockchain has promised a forthcoming ‘post-mortem’ report on the issue that will elaborate further on the facts within the next few days.
Continued issues
In his written comments, Antonopoulos expressed his thanks to the company’s consumer and business users for their patience while it worked to correct the issue.
“We understand how frustrating it can be to be unable to access funds and we would like to take every precaution necessary to ensure this does not recur. Our team has worked non-stop to ensure services are returned to you in a timely manner.”
At press time, however, Blockchain had issued another blog post detailing additional problems being experienced by some users, though they were unrelated to the larger outage.
Blockchain indicated that the company is currently working to address an issue being experienced by existing iPhone users, and that SharedCoin would continue to remain offline.
It reassured iPhone users that all funds are currently accounted for despite the glitch.
Blockchain’s response
In contrast to measures taken by other high-profile companies during outages, the Blockchain team was unusually communicative about issues, releasing five blog posts between the initial message and the final declaration that services would be restored.
In between, the company stressed to users that it was dedicating its efforts to restoring services, and reiterated to users that no funds were affected by the error.
The blockchain team was active updating users on reddit – posting multiple, live edits to messages, and in responding to frustrated users on Twitter in what amounted to an agressive PR campaign to educate and assure users.
@nawazkiani That is categorically false. No funds or accounts have been compromised. We don’t have access to your keys.
— Blockchain.info (@blockchain) March 19, 2014
Community reaction
Blockchain’s response to the ordeal was greeted positively by the digital currency community, with many lauding the willingness of the team to remain open about problems and progress.
@aantonop@blockchain Keep up the great work! A+ for giving users a way to access their funds while going through the outage!
— CryptoBucket (@CryptoBucket) March 19, 2014
Mt. Gox was roundly criticized for its silence in the wake of massive theft and its eventual insolvency. Further, what statements it did issue did little to stem frustration and caused huge sell offs among investors.
Though Blockchain’s efforts seem to have avoided such mistakes, some remained unsatisfied with the efforts.
@aantonop @blockchain 72 hours of downtime should result in a policy review and someone fired instead of tweets honestly. But, bitcoin.
— Cuke King (@cukeking) March 19, 2014
Next steps
Speaking to CoinDesk, CEO Nicolas Cary indicated that Blockchain would embark on a large-scale review of the service issues.
Cary indicated that the company would continue its policy of remaining open with users and any and all concerns.
“We’ve just started our post-mortem research and it will be a few days before we have comprehensive documentation of the incident. […] We will be publishing the entire account on our blog so the whole Bitcoin community can learn from this.”
Antonopoulos also expressed his frustration at the outage, vowing that more work would be done to restore user confidence in the coming days and weeks.
“We take pride in our work and care deeply about our customers, so this outage was humbling and painful for us too. We will do everything we can to regain your trust and make sure this doesn’t happen again.”