java.net.HttpURLConnection:large file to upload
hello,
I want to upload large file to my sever,idea:set a maxmalChunkSize ,split file in chunks ,e.g. 5M file split in 2M,2M,1M and write into outputstream, server side received the chunks of a file,and write into a empty file in order to merge all of the chunks.
but I met some problem. during the uploding, the server respons only the last chunk of the file,that is: only the last chunk of the file is uploaded,the rest chunks are not.i dont know why. Is anybody could help me,that would be nice.thanks a lot!
code:
Code :
public boolean openConnection(String numberOfFiles,String type) {
try{
this.numberOfFiles = numberOfFiles;
//this.connect = new URL("http://127.0.0.1/bdb/bdb_create.php");
this.connection=(HttpURLConnection)(this.connect.openConnection());
this.connection.setDoInput(true);
this.connection.setDoOutput(true);
this.connection.setUseCaches(false);
this.connection.setRequestMethod("POST");
this.connection.setRequestProperty("Connection", "Keep-Alive");
System.out.println("#########################OPEN CONNECTION...connected...################################");
System.out.println(type+"; boundary=******");
this.connection.setRequestProperty("Content-Type",type+";boundary=******");
this.fileOut = new DataOutputStream(this.connection.getOutputStream());
}
catch(MalformedURLException ie){
return false;
}//ie.printStackTrace();}
catch (ProtocolException ex) {
System.out.println(ex);
}
catch (IOException ioex) {
System.out.println(ioex);
}
return true;
}
private final static int BUFLEN = 4096;
private final byte readBuffer[] = new byte[BUFLEN];
public boolean submitChunks(File fileName){
try{
this.uploadRemainingLength = this.getUploadLength(fileName);
this.fileSize = this.getUploadLength(fileName);
System.out.println("=>Original filesize= "+this.fileSize);
boolean bLastChunk = false;
int chunkPart = 0;
long contentLength = 0;
long thisChunkSize = 0;
//int chunkCount = 0;
//bChunkEnabled=1
this.fileOut.writeBytes("--" + "******" + "\r\n");
this.fileOut.writeBytes("Content-Disposition: form-data;name=\"bChunkEnabled\""+"\r\n");
this.fileOut.writeBytes("\r\n");
this.fileOut.writeBytes("1");
this.fileOut.writeBytes("\r\n");
System.out.println("--" + "******" + "\r\n");
System.out.println("Content-Disposition: form-data;name=\"bChunkEnabled\""+"\r\n");
System.out.println("\r\n");
System.out.println("1");
System.out.println("\r\n");
while(!bLastChunk){
chunkPart = chunkPart +1;
bLastChunk = (contentLength > this.getRemainingLength(fileName));
if(bLastChunk){
thisChunkSize = this.getRemainingLength(fileName);
System.out.println("@=>this is the last chunk part "+chunkPart+", " +
"chunkpart size= "+thisChunkSize+" uploading...");
System.out.println("insgesamt "+chunkPart+" chunks");
}else{
thisChunkSize = this.maxChunkSize;
System.out.println("@=>this is the chunkpart "+chunkPart+", chunkpart size= "+maxChunkSize+" uploading...");
}
contentLength = thisChunkSize;
//chunkPart
this.fileOut.writeBytes("--" + "******" + "\r\n");
this.fileOut.writeBytes("Content-Disposition: form-data;name=\"chunkPart\"" + "\r\n");
this.fileOut.writeBytes("\r\n");
this.fileOut.writeBytes(Integer.toString(chunkPart));
this.fileOut.writeBytes("\r\n");
System.out.println("--" + "******" + "\r\n");
System.out.println("Content-Disposition: form-data;name=\"chunkPart\"" + "\r\n");
System.out.println("\r\n");
System.out.println(Integer.toString(chunkPart));
System.out.println("\r\n");
//bLastChunk
this.fileOut.writeBytes("--" + "******" + "\r\n");
this.fileOut.writeBytes("Content-Disposition: form-data;name=\"bLastChunk\"" + "\r\n");
this.fileOut.writeBytes("\r\n");
this.fileOut.writeBytes(String.valueOf(bLastChunk));
this.fileOut.writeBytes("\r\n");
System.out.println("--" + "******" + "\r\n");
System.out.println("Content-Disposition: form-data;name=\"bLastChunk\"" + "\r\n");
System.out.println("\r\n");
System.out.println(String.valueOf(bLastChunk));
System.out.println("\r\n");
//file data
this.fileOut.writeBytes("--" + "******" + "\r\n");
this.fileOut.writeBytes("Content-Disposition: form-data;name=\""+this.numberOfFiles+"\";"
+ " filename=\"" + fileName.toString() +"\"" + "\r\n");
this.fileOut.writeBytes("\r\n");
System.out.println("--" + "******" + "\r\n");
System.out.println("Content-Disposition: form-data;name=\""+this.numberOfFiles+"\";"
+ " filename=\"" + fileName.toString() +"\"" + "\r\n");
System.out.println("\r\n");
this.fileIn = new FileInputStream(fileName);
int amount = (int)(thisChunkSize);
while(amount > 0){
int toread = (amount > BUFLEN) ? BUFLEN : (int) amount;
int towrite = 0;
try{
towrite = this.fileIn.read(this.readBuffer,0,toread);
}catch (IOException e) {
e.printStackTrace();
}
if(towrite > 0){
try{
fileOut.write(this.readBuffer,0,towrite);
amount = amount - towrite;
this.uploadRemainingLength = this.uploadRemainingLength - towrite;
System.out.println("->write into outputstream: "+towrite
+" || the left to be uploading: "+amount);
}catch (IOException ioe) {
ioe.printStackTrace();
}
}
}
this.fileOut.writeBytes("\r\n");
this.fileOut.writeBytes("--" + "******" +"--"+"\r\n");
System.out.println("\r\n");
System.out.println("--" + "******" +"--"+"\r\n");
this.fileOut.flush();
this.fileIn.close();
}//end while
} catch (IOException ex) {ex.getStackTrace();}
BufferedReader input;
//DataInputStream input;
try {
input = new BufferedReader(new InputStreamReader(this.connection.getInputStream()));
//input = new DataInputStream(this.connection.getInputStream());
String out="";
while(out!=null) {
out = input.readLine();
System.out.println("response:"+out);
}
input.close();
} catch (IOException e) {e.printStackTrace();}
return true;
}
debug output:
Code :
#########################OPEN CONNECTION...connected...################################
multipart/form-data; boundary=******
A.filesize bigger than maxchunksize,we use chunk upload
=>Original filesize= 36192
--******
Content-Disposition: form-data;name="bChunkEnabled"
1
@=>this is the chunkpart 1, chunkpart size= 20000 uploading...
--******
Content-Disposition: form-data;name="chunkPart"
1
--******
Content-Disposition: form-data;name="bLastChunk"
false
--******
Content-Disposition: form-data;name="10"; filename="/home/i6stud/sijitang/Desktop/demo/export_bilder_10/export_bilder/5.jpg"
->write into outputstream: 4096 || the left to be uploading: 15904
->write into outputstream: 4096 || the left to be uploading: 11808
->write into outputstream: 4096 || the left to be uploading: 7712
->write into outputstream: 4096 || the left to be uploading: 3616
->write into outputstream: 3616 || the left to be uploading: 0
--******--
@=>this is the last chunk part 2, chunkpart size= 16192 uploading...
insgesamt 2 chunks
--******
Content-Disposition: form-data;name="chunkPart"
2
--******
Content-Disposition: form-data;name="bLastChunk"
true
--******
Content-Disposition: form-data;name="10"; filename="/home/i6stud/sijitang/Desktop/demo/export_bilder_10/export_bilder/5.jpg"
->write into outputstream: 4096 || the left to be uploading: 12096
->write into outputstream: 4096 || the left to be uploading: 8000
->write into outputstream: 4096 || the left to be uploading: 3904
->write into outputstream: 3904 || the left to be uploading: 0
--******--
response:<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
response: "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
response:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="de" lang="de">
response: <head>
response: <title>FAU-Bilddatenbank::FAU-Bilddatenbank</title>
response: <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
response: <meta name="Language" content="Deutsch" />
response:...
...
response:</html>
response:null
#############################DISCONNECTED##################################### 0
Re: java.net.HttpURLConnection:large file to upload
I have tried the way you mentioned to call the servlet from the client side.
But the connection is not getting established between the client and the server.
Even I have a similar requirement .... I want to upload a file greater than 10GB, where the user need to login to upload the file. He can even pause the download for days and then resume the same. And in that even the upload should start from the point paused.
We have tried with Apache Commons and the URL Method that u have mentioned. In the first case the problem that we have faced is skipping the InputStream receieved in the server. For example a file of size 10GB, has been paused after uploading of 9.5GB, now when the upload is resumed we have to skip 9.5GB and then start dumping the content of the file in the desired location.The time taken to skip the InputStream is same as uploading it back again, It is taking huge time to reach the place from where to begin the dumping is there any way we can skip the same in faster way.
Secondly, the way that u have mentioned is ur post can ease the work, but the servlet is not getting called at all if you can share some code which specifies how you are accessing the stream or is their any tweak why the servlet is not getting invoked then it would be great help for us.
thanx in advance