Internal error after a few hours

[blockcloud]

I use the api to store the bitcoin data to the relation database, but when it works for few hours(almost three), there will be an error. I don't know why this happens? Is there anyone get the save problem?

[mariosk]

@blockcloud what is the exact error? Can you provide a test with reproduction steps?

[blockcloud]

@mariosk I receive the block data through the "BlockExplorer" like this

BlockExplorer blockExplorer = new BlockExplorer();
List<Block> block = blockExplorer.getBlocksAtHeight(height);

Last few times I get the "Internal error".
This time I get the error like below.

java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:209)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
	at sun.security.ssl.InputRecord.read(InputRecord.java:532)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
	at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:552)
	at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609)
	at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:696)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3336)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at info.blockchain.api.HttpClient.inputStreamToString(HttpClient.java:162)
	at info.blockchain.api.HttpClient.openURL(HttpClient.java:108)
	at info.blockchain.api.HttpClient.get(HttpClient.java:47)
	at info.blockchain.api.blockexplorer.BlockExplorer.getBlocksAtHeight(BlockExplorer.java:134)
	at com.cp.services.GetBlocks.getBlockByHeight(GetBlocks.java:89)
	at com.cp.services.GetBlocks.getBlocksByHeight(GetBlocks.java:111)
	at com.cp.tasks.GetBlockMultiThread.run(GetBlocksInfo.java:37)
	at java.lang.Thread.run(Thread.java:745)

[blockcloud]

@mariosk There is also an error like below:

info.blockchain.api.APIException: Internal Server Error
	at info.blockchain.api.HttpClient.openURL(HttpClient.java:106)
	at info.blockchain.api.HttpClient.get(HttpClient.java:47)
	at info.blockchain.api.blockexplorer.BlockExplorer.getBlocksAtHeight(BlockExplorer.java:134)
	at com.cp.services.GetBlocks.getBlockByHeight(GetBlocks.java:88)
	at com.cp.services.GetBlocks.getBlocksByHeight(GetBlocks.java:110)
	at com.cp.tasks.GetBlocksInfo.main(GetBlocksInfo.java:140)

[mariosk]

@blockcloud it seems to me that both of the problems you see are network (blockchain.info) issues that maybe shown during high load or something. @RayXY any other ideas?

[blockcloud]

@mariosk Yeah, you are right. I want to download all the bitcoin datas from blockchain.info, but it stopped after a while every time. Do you have some suggestions? @mariosk @RayXY

[mariosk]

@blockcloud you could keep track of the blockheight you have downloaded so far and if such an error occurs, sleep for 3 seconds and retry for the last blockheight. Does it make sense?

[blockcloud]

@mariosk I have tried your suggestion, finally it works. Thank you.
Are there better ways to parse the bitcoin datas to store in a relationship database?

[mariosk]

@blockcloud I have no idea what else could be done to retrieve all the data. I suspect that this is done because the blockchain.info server has a rate limiting enabled and may drop the connections. Can you try to sleep 200ms between the requests?

[blockcloud]

@mariosk Thank you for your sugguestion. I will try it to see if it could be faster.